00010000:00010000:1.0F:1786734713.089341:0:174831:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1786734713.663413:0:174881:0:(client.c:2702:ptlrpc_set_wait()) set ffff9fc405eda0c0 going to sleep for 0 seconds 00010000:00010000:0.0F:1786734714.113278:0:174832:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1786734714.687351:0:174881:0:(client.c:2702:ptlrpc_set_wait()) set ffff9fc405eda0c0 going to sleep for 0 seconds 00000100:00080000:1.0:1786734715.008362:0:133016:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-mdtlov_UUID->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 0 suppress 0 00000100:00080000:1.0:1786734715.008369:0:133016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1786734715.008372:0:133016:0:(import.c:65:import_set_state_nolock()) ffff9fc302fa7800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1786734715.008381:0:133016:0:(import.c:510:import_select_connection()) lustre-MDT0001-osp-MDT0000: connect to NID 0@lo last attempt 5892 00000100:00080000:1.0:1786734715.008386:0:133016:0:(import.c:589:import_select_connection()) lustre-MDT0001-osp-MDT0000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1786734715.008393:0:133016:0:(import.c:624:import_select_connection()) lustre-MDT0001-osp-MDT0000: import ffff9fc302fa7800 using connection 192.168.202.117@tcp/0@lo 00000100:00100000:1.0:1786734715.008416:0:133016:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9fc302c8d880 x1873521385721344/t0(0) o38->lustre-MDT0001-osp-MDT0000@0@lo:24/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1786734715.008432:0:133016:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-mdtlov_UUID->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 0 suppress 1 00000100:00080000:1.0:1786734715.008436:0:133016:0:(pinger.c:247:ptlrpc_pinger_process_import()) lustre-MDT0000-mdtlov_UUID->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1786734715.008439:0:133016:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-mdtlov_UUID->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 0 suppress 1 00000100:00080000:1.0:1786734715.008442:0:133016:0:(pinger.c:247:ptlrpc_pinger_process_import()) lustre-MDT0000-mdtlov_UUID->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1786734715.008793:0:16418:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9fc302c8e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-MDT0000_UUID:16418:1873521385721472:0@lo:400:kworker.0 00000100:00100000:1.0:1786734715.008899:0:16418:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1786734715.008986:0:16418:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1786734715.008993:0:16418:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9fc302c8ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:187aa59a-98ad-4057-896e-cfade62d2452:16418:1873521385721216:0@lo:400:kworker.0 00000100:00100000:1.0:1786734715.009037:0:16418:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1786734715.009079:0:174838:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1873521385721216 00000100:00100000:1.0:1786734715.009086:0:174838:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1873521385721216 00000100:00100000:1.0:1786734715.009146:0:174838:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 28 00000100:00100000:1.0:1786734715.009156:0:174838:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9fc303179c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:187aa59a-98ad-4057-896e-cfade62d2452+12:16418:x1873521385721216:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1786734715.009208:0:174838:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9fc303179c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:187aa59a-98ad-4057-896e-cfade62d2452+12:16418:x1873521385721216:12345-0@lo:400:kworker.0 Request processed in 54us (170us total) trans 0 rc 0/0 00000100:00100000:1.0:1786734715.009257:0:174838:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 28 00000100:00100000:1.0:1786734715.009304:0:16416:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9fc302c8d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-mdtlov_UUID:16416:1873521385721344:0@lo:38:kworker.0 00000100:00100000:1.0:1786734715.009331:0:16416:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1786734715.009359:0:174848:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1873521385721344 00000100:00100000:1.0:1786734715.009362:0:174848:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1873521385721344 00000100:00100000:1.0:1786734715.009379:0:174848:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 4 00000100:00100000:1.0:1786734715.009383:0:174848:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9fc302c8c700 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:0+-99:16416:x1873521385721344:12345-0@lo:38:kworker.0 00010000:00020000:1.0:1786734715.009399:0:174848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. 00000100:00100000:1.0:1786734715.009426:0:174848:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9fc302c8c700 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:0+-99:16416:x1873521385721344:12345-0@lo:38:kworker.0 Request processed in 41us (95us total) trans 0 rc -19/-19 00000100:00100000:1.0:1786734715.009432:0:174848:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 4 00000100:00080000:1.0:1786734715.009463:0:16416:0:(import.c:65:import_set_state_nolock()) ffff9fc302fa7800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1786734715.009468:0:16416:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 0@lo failed (-19) 00000100:00100000:1.0:1786734715.009471:0:16416:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9fc302c8d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-mdtlov_UUID:16416:1873521385721344:0@lo:38:kworker.0 00000100:00100000:1.0:1786734715.010709:0:16418:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9fc302c8ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:187aa59a-98ad-4057-896e-cfade62d2452:16418:1873521385721216:0@lo:400:kworker.0 00000100:00100000:0.0:1786734715.011293:0:174844:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1873521385721472 00000100:00100000:0.0:1786734715.011301:0:174844:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1873521385721472 00000100:00100000:0.0:1786734715.011345:0:174844:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 2 00000100:00100000:0.0:1786734715.011357:0:174844:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9fc302c8e680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0000-lwp-MDT0000_UUID+5:16418:x1873521385721472:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1786734715.011433:0:174844:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9fc302c8e680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0000-lwp-MDT0000_UUID+5:16418:x1873521385721472:12345-0@lo:400:kworker.0 Request processed in 81us (2535us total) trans 0 rc 0/0 00000100:00100000:0.0:1786734715.011440:0:174844:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 2 00010000:00080000:1.0:1786734715.011619:0:16418:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=184649) req@ffff9fc302c8e300 x1873521385721472/t0(0) o400->lustre-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1786734731 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1786734715.011631:0:16418:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9fc302c8e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-MDT0000_UUID:16418:1873521385721472:0@lo:400:kworker.0 00010000:00010000:0.0:1786734715.138402:0:174831:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0F:1786734715.717446:0:174881:0:(client.c:2702:ptlrpc_set_wait()) set ffff9fc405eda0c0 going to sleep for 0 seconds 00010000:00010000:2.0F:1786734716.160260:0:174832:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1786734716.739486:0:174881:0:(client.c:2702:ptlrpc_set_wait()) set ffff9fc405eda0c0 going to sleep for 0 seconds 00010000:00010000:0.0:1786734717.183841:0:174831:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1786734717.759986:0:174881:0:(client.c:2702:ptlrpc_set_wait()) set ffff9fc405eda0c0 going to sleep for 0 seconds 00010000:00010000:2.0:1786734718.207577:0:174832:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1786734718.784309:0:174881:0:(client.c:2702:ptlrpc_set_wait()) set ffff9fc405eda0c0 going to sleep for 0 seconds 00000100:00100000:2.0:1786734719.103442:0:16416:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786734704/real 1786734704] req@ffff9fc302cd5880 x1873521385720448/t0(0) o8->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 1 dl 1786734719 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1786734719.103499:0:16416:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9fc302cd5880 x1873521385720448/t0(0) o8->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 1 dl 1786734719 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1786734719.103512:0:16416:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786734704/real 1786734704] req@ffff9fc302cd4700 x1873521385720576/t0(0) o8->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 1 dl 1786734719 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1786734719.103691:0:16416:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9fc302cd4700 x1873521385720576/t0(0) o8->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 1 dl 1786734719 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1786734719.103717:0:16416:0:(import.c:65:import_set_state_nolock()) ffff9fc3042a4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1786734719.103853:0:16416:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 0@lo failed (-110) 00000100:00100000:2.0:1786734719.103859:0:16416:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9fc302cd5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-mdtlov_UUID:16416:1873521385720448:0@lo:8:kworker.0 00000100:00080000:2.0:1786734719.103867:0:16416:0:(import.c:65:import_set_state_nolock()) ffff9fc405dec000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1786734719.104298:0:164651:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-mdtlov_UUID->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 0 suppress 0 00000100:00080000:0.0:1786734719.104305:0:164651:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1786734719.104308:0:164651:0:(import.c:65:import_set_state_nolock()) ffff9fc302fa7800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1786734719.104316:0:164651:0:(import.c:510:import_select_connection()) lustre-MDT0001-osp-MDT0000: connect to NID 0@lo last attempt 5897 00000100:00080000:0.0:1786734719.104323:0:164651:0:(import.c:589:import_select_connection()) lustre-MDT0001-osp-MDT0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1786734719.104334:0:164651:0:(import.c:624:import_select_connection()) lustre-MDT0001-osp-MDT0000: import ffff9fc302fa7800 using connection 192.168.202.117@tcp/0@lo 00000100:00100000:0.0:1786734719.104371:0:164651:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9fc302cd6680 x1873521385721728/t0(0) o38->lustre-MDT0001-osp-MDT0000@0@lo:24/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1786734719.104383:0:164651:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-mdtlov_UUID->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 0 suppress 1 00000100:00080000:0.0:1786734719.104386:0:164651:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1786734719.104388:0:164651:0:(import.c:65:import_set_state_nolock()) ffff9fc3042a4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1786734719.104392:0:164651:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-MDT0000: connect to NID 0@lo last attempt 5886 00000100:00080000:0.0:1786734719.104396:0:164651:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-MDT0000: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1786734719.104400:0:164651:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-MDT0000: import ffff9fc3042a4800 using connection 192.168.202.117@tcp/0@lo 00000100:00100000:0.0:1786734719.104414:0:164651:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9fc302cd7480 x1873521385721856/t0(0) o8->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1786734719.104421:0:164651:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-mdtlov_UUID->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 0 suppress 1 00000100:00080000:0.0:1786734719.104444:0:164651:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1786734719.104447:0:164651:0:(import.c:65:import_set_state_nolock()) ffff9fc405dec000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1786734719.104451:0:164651:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-MDT0000: connect to NID 0@lo last attempt 5886 00000100:00080000:0.0:1786734719.104454:0:164651:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-MDT0000: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1786734719.104458:0:164651:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-MDT0000: import ffff9fc405dec000 using connection 192.168.202.117@tcp/0@lo 00000100:00100000:0.0:1786734719.104473:0:164651:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9fc302cd5500 x1873521385721984/t0(0) o8->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1786734719.104537:0:16420:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9fc302cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-MDT0000_UUID:16420:1873521385722112:0@lo:400:kworker.0 00000100:00100000:0.0:1786734719.104669:0:16420:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1786734719.104730:0:16420:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:0.0:1786734719.104735:0:16420:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9fc302cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:187aa59a-98ad-4057-896e-cfade62d2452:16420:1873521385721600:0@lo:400:kworker.0 00000100:00100000:0.0:1786734719.104778:0:16420:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1786734719.104824:0:174837:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1873521385721600 00000100:00100000:0.0:1786734719.104834:0:174837:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1873521385721600 00000100:00100000:0.0:1786734719.104888:0:174837:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 29 00000100:00100000:0.0:1786734719.104911:0:174837:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9fc30317b450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:187aa59a-98ad-4057-896e-cfade62d2452+12:16420:x1873521385721600:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1786734719.104984:0:174837:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9fc30317b450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:187aa59a-98ad-4057-896e-cfade62d2452+12:16420:x1873521385721600:12345-0@lo:400:kworker.0 Request processed in 84us (203us total) trans 0 rc 0/0 00000100:00100000:0.0:1786734719.104992:0:174837:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 29 00000100:00100000:0.0:1786734719.105060:0:16420:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9fc302cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:187aa59a-98ad-4057-896e-cfade62d2452:16420:1873521385721600:0@lo:400:kworker.0 00000100:00100000:1.0:1786734719.105507:0:174843:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1873521385722112 00000100:00100000:1.0:1786734719.105513:0:174843:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1873521385722112 00000100:00100000:1.0:1786734719.105566:0:174843:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 3 00000100:00100000:1.0:1786734719.105573:0:174843:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9fc302cd6a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-MDT0000_UUID+5:16420:x1873521385722112:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1786734719.105630:0:174843:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9fc302cd6a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-MDT0000_UUID+5:16420:x1873521385722112:12345-0@lo:400:kworker.0 Request processed in 58us (962us total) trans 0 rc 0/0 00000100:00100000:1.0:1786734719.105657:0:174843:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 3 00010000:00080000:1.0:1786734719.105700:0:16420:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=184649) req@ffff9fc302cd7100 x1873521385722112/t0(0) o400->lustre-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1786734735 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1786734719.105713:0:16420:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9fc302cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-MDT0000_UUID:16420:1873521385722112:0@lo:400:kworker.0 00000100:00080000:2.0:1786734719.105972:0:16416:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 0@lo failed (-110) 00000100:00100000:2.0:1786734719.105979:0:16416:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9fc302cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-mdtlov_UUID:16416:1873521385720576:0@lo:8:kworker.0 00000100:00100000:2.0:1786734719.106004:0:16416:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9fc302cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-mdtlov_UUID:16416:1873521385721728:0@lo:38:kworker.0 00000100:00100000:2.0:1786734719.106143:0:16416:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1786734719.106170:0:16416:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9fc302cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-mdtlov_UUID:16416:1873521385721856:0@lo:8:kworker.0 00000400:00000100:2.0:1786734719.106197:0:16416:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-0@lo portal 28 match 1873521385721856 offset 0 length 520: 4 00000100:00100000:2.0:1786734719.106237:0:16416:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9fc302cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-mdtlov_UUID:16416:1873521385721984:0@lo:8:kworker.0 00000400:00000100:2.0:1786734719.106259:0:16416:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-0@lo portal 28 match 1873521385721984 offset 0 length 520: 4 00000100:00100000:1.0:1786734719.106379:0:174848:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1873521385721728 00000100:00100000:1.0:1786734719.106384:0:174848:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1873521385721728 00000100:00100000:1.0:1786734719.106419:0:174848:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 5 00000100:00100000:1.0:1786734719.106426:0:174848:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9fc43f03d180 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:0+-99:16416:x1873521385721728:12345-0@lo:38:kworker.0 00010000:00020000:1.0:1786734719.106447:0:174848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. 00000100:00100000:1.0:1786734719.106617:0:174848:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9fc43f03d180 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_000:0+-99:16416:x1873521385721728:12345-0@lo:38:kworker.0 Request processed in 190us (474us total) trans 0 rc -19/-19 00000100:00100000:1.0:1786734719.106625:0:174848:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 5 00000100:00080000:0.0:1786734719.110363:0:16416:0:(import.c:65:import_set_state_nolock()) ffff9fc302fa7800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1786734719.110382:0:16416:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1786734719.110388:0:16416:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9fc302cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-mdtlov_UUID:16416:1873521385721728:0@lo:38:kworker.0 00010000:00010000:1.0:1786734719.235650:0:174831:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1786734719.751937:0:175715:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == sanity-scrub test 24: object iteration copes with objects created in the middle of it ========================================================== 15:11:55 (1786734715) 00000100:00100000:2.0:1786734719.807462:0:174881:0:(client.c:2702:ptlrpc_set_wait()) set ffff9fc405eda0c0 going to sleep for 0 seconds 00010000:00010000:2.0:1786734720.256749:0:174832:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1786734720.831332:0:174881:0:(client.c:2702:ptlrpc_set_wait()) set ffff9fc405eda0c0 going to sleep for 0 seconds 00010000:00010000:1.0:1786734721.279522:0:174831:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1786734721.856158:0:174881:0:(client.c:2702:ptlrpc_set_wait()) set ffff9fc405eda0c0 going to sleep for 0 seconds 00010000:00010000:2.0:1786734722.303833:0:174832:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1786734722.887042:0:174881:0:(client.c:2702:ptlrpc_set_wait()) set ffff9fc405eda0c0 going to sleep for 0 seconds 00000001:02000400:3.0:1786734723.079380:0:175814:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: sanity-scrub test_24: @@@@@@ FAIL: failed to create /mnt/lustre/d24 00010000:00010000:3.0:1786734723.333291:0:174831:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1786734723.903357:0:174881:0:(client.c:2702:ptlrpc_set_wait()) set ffff9fc405eda0c0 going to sleep for 0 seconds 00000100:00080000:2.0:1786734724.223960:0:133016:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-mdtlov_UUID->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 0 suppress 0 00000100:00080000:2.0:1786734724.223972:0:133016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1786734724.223977:0:133016:0:(import.c:65:import_set_state_nolock()) ffff9fc302fa7800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1786734724.223991:0:133016:0:(import.c:510:import_select_connection()) lustre-MDT0001-osp-MDT0000: connect to NID 0@lo last attempt 5901 00000100:00080000:2.0:1786734724.223998:0:133016:0:(import.c:589:import_select_connection()) lustre-MDT0001-osp-MDT0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1786734724.224008:0:133016:0:(import.c:624:import_select_connection()) lustre-MDT0001-osp-MDT0000: import ffff9fc302fa7800 using connection 192.168.202.117@tcp/0@lo 00000100:00100000:2.0:1786734724.224043:0:133016:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9fc43f03dc00 x1873521385722368/t0(0) o38->lustre-MDT0001-osp-MDT0000@0@lo:24/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1786734724.224067:0:133016:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-mdtlov_UUID->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 0 suppress 1 00000100:00080000:2.0:1786734724.224071:0:133016:0:(pinger.c:247:ptlrpc_pinger_process_import()) lustre-MDT0000-mdtlov_UUID->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1786734724.224075:0:133016:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-mdtlov_UUID->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 0 suppress 1 00000100:00080000:2.0:1786734724.224078:0:133016:0:(pinger.c:247:ptlrpc_pinger_process_import()) lustre-MDT0000-mdtlov_UUID->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1786734724.224167:0:16418:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9fc43f03d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-MDT0000_UUID:16418:1873521385722496:0@lo:400:kworker.0 00000100:00100000:1.0:1786734724.224296:0:16418:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1786734724.224382:0:16418:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1786734724.224389:0:16418:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9fc43f03df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:187aa59a-98ad-4057-896e-cfade62d2452:16418:1873521385722240:0@lo:400:kworker.0 00000100:00100000:1.0:1786734724.224423:0:16418:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1786734724.224468:0:174836:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1873521385722240 00000100:00100000:1.0:1786734724.224478:0:174836:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1873521385722240 00000100:00100000:1.0:1786734724.224544:0:174836:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 30 00000100:00100000:1.0:1786734724.224557:0:174836:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9fc30317ac50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:187aa59a-98ad-4057-896e-cfade62d2452+12:16418:x1873521385722240:12345-0@lo:400:kworker.0 00000100:00100000:1.0:1786734724.224633:0:174836:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9fc30317ac50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:187aa59a-98ad-4057-896e-cfade62d2452+12:16418:x1873521385722240:12345-0@lo:400:kworker.0 Request processed in 77us (207us total) trans 0 rc 0/0 00000100:00100000:1.0:1786734724.224661:0:174836:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 30 00000100:00100000:1.0:1786734724.224720:0:16418:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9fc43f03df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:187aa59a-98ad-4057-896e-cfade62d2452:16418:1873521385722240:0@lo:400:kworker.0 00000100:00100000:2.0:1786734724.225139:0:174845:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1873521385722496 00000100:00100000:2.0:1786734724.225144:0:174845:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1873521385722496 00000100:00100000:2.0:1786734724.225187:0:174845:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 4 00000100:00100000:2.0:1786734724.225196:0:174845:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9fc302c8e300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-MDT0000_UUID+5:16418:x1873521385722496:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1786734724.225267:0:174845:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9fc302c8e300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:lustre-MDT0000-lwp-MDT0000_UUID+5:16418:x1873521385722496:12345-0@lo:400:kworker.0 Request processed in 73us (972us total) trans 0 rc 0/0 00000100:00100000:2.0:1786734724.225275:0:174845:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 4 00010000:00080000:2.0:1786734724.225307:0:16418:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=184649) req@ffff9fc43f03d500 x1873521385722496/t0(0) o400->lustre-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1786734740 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1786734724.225341:0:16418:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9fc43f03d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-MDT0000_UUID:16418:1873521385722496:0@lo:400:kworker.0 00000100:00100000:0.0:1786734724.225482:0:16416:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9fc43f03dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-mdtlov_UUID:16416:1873521385722368:0@lo:38:kworker.0 00000100:00100000:0.0:1786734724.225537:0:16416:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1786734724.225740:0:174849:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1873521385722368 00000100:00100000:2.0:1786734724.225746:0:174849:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1873521385722368 00000100:00100000:2.0:1786734724.225992:0:174849:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 6 00000100:00100000:2.0:1786734724.225998:0:174849:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9fc302cd6680 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:0+-99:16416:x1873521385722368:12345-0@lo:38:kworker.0 00010000:00020000:2.0:1786734724.226020:0:174849:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. 00000100:00100000:2.0:1786734724.226074:0:174849:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9fc302cd6680 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:0+-99:16416:x1873521385722368:12345-0@lo:38:kworker.0 Request processed in 75us (537us total) trans 0 rc -19/-19 00000100:00100000:2.0:1786734724.226081:0:174849:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 6 00000100:00080000:0.0:1786734724.226160:0:16416:0:(import.c:65:import_set_state_nolock()) ffff9fc302fa7800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1786734724.226168:0:16416:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1786734724.226171:0:16416:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9fc43f03dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-mdtlov_UUID:16416:1873521385722368:0@lo:38:kworker.0 00010000:00010000:1.0:1786734724.354841:0:174832:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 178 lines, 178 kept, 0 dropped, 0 bad.