00000100:00100000:2.0F:1786734731.080639:0:174881:0:(client.c:2702:ptlrpc_set_wait()) set ffff9fc405eda0c0 going to sleep for 0 seconds 00010000:00010000:2.0:1786734731.519492:0:174831:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1786734732.095331:0:174881:0:(client.c:2702:ptlrpc_set_wait()) set ffff9fc405eda0c0 going to sleep for 0 seconds 00010000:00010000:3.0F:1786734732.545527:0:174832:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1786734733.121033:0:174881:0:(client.c:2702:ptlrpc_set_wait()) set ffff9fc405eda0c0 going to sleep for 0 seconds 00010000:00010000:1.0F:1786734733.571190:0:174831:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1786734734.143444:0:174881:0:(client.c:2702:ptlrpc_set_wait()) set ffff9fc405eda0c0 going to sleep for 0 seconds 00000100:00080000:3.0:1786734734.474915:0:133016:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-mdtlov_UUID->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 0 suppress 0 00000100:00080000:3.0:1786734734.474924:0:133016:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1786734734.474929:0:133016:0:(import.c:65:import_set_state_nolock()) ffff9fc302fa7800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1786734734.474939:0:133016:0:(import.c:510:import_select_connection()) lustre-MDT0001-osp-MDT0000: connect to NID 0@lo last attempt 5911 00000100:00080000:3.0:1786734734.474946:0:133016:0:(import.c:589:import_select_connection()) lustre-MDT0001-osp-MDT0000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1786734734.474956:0:133016:0:(import.c:624:import_select_connection()) lustre-MDT0001-osp-MDT0000: import ffff9fc302fa7800 using connection 192.168.202.117@tcp/0@lo 00000100:00100000:3.0:1786734734.474985:0:133016:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9fc43d8a3800 x1873521385723136/t0(0) o38->lustre-MDT0001-osp-MDT0000@0@lo:24/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0F:1786734734.475053:0:16416:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9fc43d8a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-mdtlov_UUID:16416:1873521385723136:0@lo:38:kworker.0 00000100:00100000:2.0:1786734734.475107:0:16417:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9fc43d8a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:187aa59a-98ad-4057-896e-cfade62d2452:16417:1873521385723008:0@lo:400:kworker.0 00000100:00080000:3.0:1786734734.475185:0:133016:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-mdtlov_UUID->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 0 suppress 1 00000100:00080000:3.0:1786734734.475190:0:133016:0:(pinger.c:247:ptlrpc_pinger_process_import()) lustre-MDT0000-mdtlov_UUID->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1786734734.475193:0:133016:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-mdtlov_UUID->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 0 suppress 1 00000100:00080000:3.0:1786734734.475197:0:133016:0:(pinger.c:247:ptlrpc_pinger_process_import()) lustre-MDT0000-mdtlov_UUID->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1786734734.475435:0:16418:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9fc43d8a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-MDT0000_UUID:16418:1873521385723264:0@lo:400:kworker.0 00000100:00100000:2.0:1786734734.477969:0:16417:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1786734734.478212:0:16416:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1786734734.478268:0:174849:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1873521385723136 00000100:00100000:0.0:1786734734.478280:0:174849:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1873521385723136 00000100:00100000:0.0:1786734734.478321:0:174849:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 8 00000100:00100000:0.0:1786734734.478330:0:174849:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9fc302cd7800 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:0+-99:16416:x1873521385723136:12345-0@lo:38:kworker.0 00010000:00020000:0.0:1786734734.478356:0:174849:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. 00000100:00100000:0.0:1786734734.478411:0:174849:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9fc302cd7800 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:0+-99:16416:x1873521385723136:12345-0@lo:38:kworker.0 Request processed in 81us (199us total) trans 0 rc -19/-19 00000100:00100000:0.0:1786734734.478419:0:174849:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 8 00000100:00080000:0.0:1786734734.478461:0:16416:0:(import.c:65:import_set_state_nolock()) ffff9fc302fa7800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1786734734.478467:0:16416:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1786734734.478471:0:16416:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9fc43d8a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-mdtlov_UUID:16416:1873521385723136:0@lo:38:kworker.0 00000100:00100000:2.0:1786734734.479212:0:174837:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1873521385723008 00000100:00100000:2.0:1786734734.479235:0:174837:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1873521385723008 00000100:00100000:2.0:1786734734.479297:0:174837:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 32 00000100:00100000:2.0:1786734734.479307:0:174837:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9fc43e892050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:187aa59a-98ad-4057-896e-cfade62d2452+12:16417:x1873521385723008:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1786734734.479376:0:174837:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9fc43e892050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:187aa59a-98ad-4057-896e-cfade62d2452+12:16417:x1873521385723008:12345-0@lo:400:kworker.0 Request processed in 72us (1420us total) trans 0 rc 0/0 00000100:00100000:2.0:1786734734.479384:0:174837:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 32 00000100:00100000:2.0:1786734734.479439:0:16417:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9fc43d8a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:187aa59a-98ad-4057-896e-cfade62d2452:16417:1873521385723008:0@lo:400:kworker.0 00000100:00100000:1.0:1786734734.486851:0:16418:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1786734734.498038:0:174843:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1873521385723264 00000100:00100000:2.0:1786734734.498064:0:174843:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1873521385723264 00000100:00100000:3.0:1786734734.510323:0:174843:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 6 00000100:00100000:3.0:1786734734.510344:0:174843:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9fc302c8ce00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-MDT0000_UUID+5:16418:x1873521385723264:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1786734734.510480:0:174843:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9fc302c8ce00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-MDT0000_UUID+5:16418:x1873521385723264:12345-0@lo:400:kworker.0 Request processed in 142us (23647us total) trans 0 rc 0/0 00000100:00100000:3.0:1786734734.510489:0:174843:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 6 00010000:00080000:1.0:1786734734.510723:0:16418:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=184649) req@ffff9fc43d8a0380 x1873521385723264/t0(0) o400->lustre-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 0 dl 1786734750 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1786734734.510743:0:16418:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9fc43d8a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-MDT0000_UUID:16418:1873521385723264:0@lo:400:kworker.0 00010000:00010000:3.0:1786734734.592187:0:174832:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1786734735.167409:0:174881:0:(client.c:2702:ptlrpc_set_wait()) set ffff9fc405eda0c0 going to sleep for 0 seconds 00010000:00010000:1.0:1786734735.615913:0:174831:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1786734736.075117:0:176259:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == sanity-scrub test 26: lctl iterate_objects on MDT (regular files only) ========================================================== 15:12:12 (1786734732) 00000100:00100000:2.0:1786734736.232270:0:174881:0:(client.c:2702:ptlrpc_set_wait()) set ffff9fc405eda0c0 going to sleep for 0 seconds 00010000:00010000:3.0:1786734736.639877:0:174832:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1786734737.282389:0:174881:0:(client.c:2702:ptlrpc_set_wait()) set ffff9fc405eda0c0 going to sleep for 0 seconds 00010000:00010000:1.0:1786734737.664840:0:174831:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1786734737.700076:0:176307:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: sanity-scrub test_26: @@@@@@ FAIL: failed to set DoM layout 00000100:00100000:3.0:1786734738.305021:0:174881:0:(client.c:2702:ptlrpc_set_wait()) set ffff9fc405eda0c0 going to sleep for 0 seconds 00010000:00010000:3.0:1786734738.691865:0:174832:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 60 lines, 60 kept, 0 dropped, 0 bad.