00000100:00100000:1.0F:1715208059.034199:0:3677:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.23@tcp (source: 12345-192.168.201.23@tcp) 00000100:00100000:1.0:1715208059.034255:0:3677:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.23@tcp (source: 12345-192.168.201.23@tcp) 00000100:00100000:1.0:1715208059.034272:0:3677:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.23@tcp (source: 12345-192.168.201.23@tcp) 00000100:00100000:1.0:1715208059.034285:0:3677:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.23@tcp (source: 12345-192.168.201.23@tcp) 00000100:00100000:2.0F:1715208059.034301:0:21028:0:(service.c:2096:ptlrpc_server_handle_req_in()) unwrap req x1798523505065472 00000100:00100000:1.0:1715208059.034317:0:3677:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.23@tcp (source: 12345-192.168.201.23@tcp) 00000100:00100000:2.0:1715208059.034324:0:21028:0:(service.c:2161:ptlrpc_server_handle_req_in()) got req x1798523505065472 00000100:00100000:0.0F:1715208059.034335:0:6554:0:(service.c:2096:ptlrpc_server_handle_req_in()) unwrap req x1798523505065600 00000100:00100000:1.0:1715208059.034346:0:17457:0:(service.c:2096:ptlrpc_server_handle_req_in()) unwrap req x1798523505065664 00000100:00100000:1.0:1715208059.034353:0:17457:0:(service.c:2161:ptlrpc_server_handle_req_in()) got req x1798523505065664 00000100:00100000:0.0:1715208059.034353:0:6554:0:(service.c:2161:ptlrpc_server_handle_req_in()) got req x1798523505065600 00000100:00100000:2.0:1715208059.034385:0:21028:0:(service.c:2096:ptlrpc_server_handle_req_in()) unwrap req x1798523505065536 00000100:00100000:2.0:1715208059.034389:0:21028:0:(service.c:2161:ptlrpc_server_handle_req_in()) got req x1798523505065536 00000100:00100000:0.0:1715208059.034398:0:6554:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.23@tcp, seq: 4354 00000100:00100000:1.0:1715208059.034400:0:17457:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.23@tcp, seq: 2332 00000100:00100000:1.0:1715208059.034408:0:17457:0:(service.c:2335:ptlrpc_server_handle_request()) Handling RPC req@ffff88009dd40e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_006:1bc50a3e-f07c-4b55-83c3-5f3c3d406528+12:2002:x1798523505065664:12345-192.168.201.23@tcp:400:kworker.0 00000100:00100000:0.0:1715208059.034408:0:6554:0:(service.c:2335:ptlrpc_server_handle_request()) Handling RPC req@ffff88009dd40700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:04532c2b-e5e9-420d-abb0-4480aab1e378+8:2003:x1798523505065472:12345-192.168.201.23@tcp:400:kworker.0 00000100:00100000:2.0:1715208059.034425:0:21028:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.23@tcp, seq: 4355 00000100:00100000:2.0:1715208059.034438:0:21028:0:(service.c:2335:ptlrpc_server_handle_request()) Handling RPC req@ffff88009dd41f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:1bc50a3e-f07c-4b55-83c3-5f3c3d406528+20:2003:x1798523505065600:12345-192.168.201.23@tcp:400:kworker.0 00000100:00100000:1.0:1715208059.034475:0:17457:0:(service.c:2384:ptlrpc_server_handle_request()) Handled RPC req@ffff88009dd40e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_006:1bc50a3e-f07c-4b55-83c3-5f3c3d406528+12:2002:x1798523505065664:12345-192.168.201.23@tcp:400:kworker.0 Request processed in 68us (157us total) trans 0 rc 0/0 00000100:00100000:0.0:1715208059.034475:0:6554:0:(service.c:2384:ptlrpc_server_handle_request()) Handled RPC req@ffff88009dd40700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:04532c2b-e5e9-420d-abb0-4480aab1e378+8:2003:x1798523505065472:12345-192.168.201.23@tcp:400:kworker.0 Request processed in 70us (286us total) trans 0 rc 0/0 00000100:00100000:1.0:1715208059.034482:0:17457:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.23@tcp, seq: 2332 00000100:00100000:0.0:1715208059.034483:0:6554:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.23@tcp, seq: 4354 00000100:00100000:0.0:1715208059.034499:0:6554:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.23@tcp, seq: 4356 00000100:00100000:0.0:1715208059.034504:0:6554:0:(service.c:2335:ptlrpc_server_handle_request()) Handling RPC req@ffff88009dd43100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:1bc50a3e-f07c-4b55-83c3-5f3c3d406528+37:2002:x1798523505065536:12345-192.168.201.23@tcp:400:kworker.0 00000100:00100000:0.0:1715208059.034526:0:6554:0:(service.c:2384:ptlrpc_server_handle_request()) Handled RPC req@ffff88009dd43100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:1bc50a3e-f07c-4b55-83c3-5f3c3d406528+37:2002:x1798523505065536:12345-192.168.201.23@tcp:400:kworker.0 Request processed in 21us (239us total) trans 0 rc 0/0 00000100:00100000:2.0:1715208059.034528:0:21028:0:(service.c:2384:ptlrpc_server_handle_request()) Handled RPC req@ffff88009dd41f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:1bc50a3e-f07c-4b55-83c3-5f3c3d406528+20:2003:x1798523505065600:12345-192.168.201.23@tcp:400:kworker.0 Request processed in 90us (250us total) trans 0 rc 0/0 00000100:00100000:0.0:1715208059.034531:0:6554:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.23@tcp, seq: 4356 00000100:00100000:2.0:1715208059.034540:0:21028:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.23@tcp, seq: 4355 00000100:00080000:0.0:1715208059.034546:0:7120:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 older than 1715208029 00000100:00080000:0.0:1715208059.034549:0:7120:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 older than 1715208029 00000100:00080000:0.0:1715208059.034551:0:7120:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0001 older than 1715208029 00000100:00100000:2.0:1715208059.034580:0:7118:0:(service.c:2096:ptlrpc_server_handle_req_in()) unwrap req x1798523505065408 00000100:00100000:2.0:1715208059.034586:0:7118:0:(service.c:2161:ptlrpc_server_handle_req_in()) got req x1798523505065408 00000100:00080000:0.0:1715208059.034629:0:7120:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd MGS older than 1715208029 00000100:00100000:2.0:1715208059.034654:0:7118:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.23@tcp, seq: 1465 00000100:00100000:2.0:1715208059.034662:0:7118:0:(service.c:2335:ptlrpc_server_handle_request()) Handling RPC req@ffff880087dfc050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:7e7a1b67-d841-4c3c-9f35-ded8f965b8dd+12:2002:x1798523505065408:12345-192.168.201.23@tcp:400:kworker.0 00000100:00100000:2.0:1715208059.034723:0:7118:0:(service.c:2384:ptlrpc_server_handle_request()) Handled RPC req@ffff880087dfc050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:7e7a1b67-d841-4c3c-9f35-ded8f965b8dd+12:2002:x1798523505065408:12345-192.168.201.23@tcp:400:kworker.0 Request processed in 59us (464us total) trans 0 rc 0/0 00000100:00100000:2.0:1715208059.034733:0:7118:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.23@tcp, seq: 1465 00010000:00010000:3.0F:1715208059.352189:0:14811:0:(ldlm_lockd.c:2860:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1715208059.791545:0:6652:0:(debug.c:705:libcfs_debug_mark_buffer()) DEBUG MARKER: sanity-hsm test_24g: @@@@@@ FAIL: hsm flags on /mnt/lustre/d24g.sanity-hsm/f24g.sanity-hsm are Dumping lctl log to /tmp/testlogs//sanity-hsm.test_24g.*.1715208058.log != 0x00000009 00000100:00100000:3.0:1715208060.308056:0:3686:0:(ptlrpcd.c:417:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:0.0:1715208060.308062:0:3684:0:(client.c:1776:ptlrpc_send_new_req()) Sending RPC req@ffff8800a0219180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3684:1798523510683904:0@lo:13:osp-pre-0-0.0 00000100:00100000:3.0:1715208060.308105:0:3686:0:(client.c:1776:ptlrpc_send_new_req()) Sending RPC req@ffff88009e03d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3686:1798523510683968:0@lo:41:osp-pre-1-0.0 00000100:00100000:0.0:1715208060.308173:0:3684:0:(events.c:359:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1715208060.308183:0:3686:0:(events.c:359:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1715208060.308220:0:9919:0:(service.c:2096:ptlrpc_server_handle_req_in()) unwrap req x1798523510683904 00000100:00100000:2.0:1715208060.308234:0:9919:0:(service.c:2161:ptlrpc_server_handle_req_in()) got req x1798523510683904 00000100:00100000:0.0:1715208060.308240:0:8276:0:(service.c:2096:ptlrpc_server_handle_req_in()) unwrap req x1798523510683968 00000100:00100000:0.0:1715208060.308248:0:8276:0:(service.c:2161:ptlrpc_server_handle_req_in()) got req x1798523510683968 00000100:00100000:0.0:1715208060.308293:0:8276:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 2000 00000100:00100000:2.0:1715208060.308299:0:9919:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 1872 00000100:00100000:0.0:1715208060.308302:0:8276:0:(service.c:2335:ptlrpc_server_handle_request()) Handling RPC req@ffff88009e03e680 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3686:x1798523510683968:12345-0@lo:41:osp-pre-1-0.0 00000100:00100000:2.0:1715208060.308308:0:9919:0:(service.c:2335:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a0219880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3684:x1798523510683904:12345-0@lo:13:osp-pre-0-0.0 00000100:00080000:3.0:1715208060.308331:0:7120:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 older than 1715208030 00000004:00000024:0.0:1715208060.308343:0:8276:0:(mdt_handler.c:569:mdt_statfs()) blocks cached 0 granted 4292608 pending 0 free 1446285312 avail 1316823040 00002000:00000024:2.0:1715208060.308347:0:9919:0:(ofd_obd.c:743:ofd_statfs()) blocks cached 0 granted 278208 pending 0 free 3922501632 avail 3690979328 00000100:00100000:0.0:1715208060.308400:0:8276:0:(service.c:2384:ptlrpc_server_handle_request()) Handled RPC req@ffff88009e03e680 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_002:lustre-MDT0000-mdtlov_UUID+5:3686:x1798523510683968:12345-0@lo:41:osp-pre-1-0.0 Request processed in 98us (218us total) trans 0 rc 0/0 00000100:00100000:0.0:1715208060.308408:0:8276:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 2000 00000100:00100000:2.0:1715208060.308428:0:9919:0:(service.c:2384:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a0219880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3684:x1798523510683904:12345-0@lo:13:osp-pre-0-0.0 Request processed in 120us (255us total) trans 0 rc 0/0 00000100:00100000:3.0:1715208060.308435:0:3686:0:(client.c:2261:ptlrpc_check_set()) Completed RPC req@ffff88009e03d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3686:1798523510683968:0@lo:41:osp-pre-1-0.0 00000100:00100000:2.0:1715208060.308437:0:9919:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 1872 00000100:00100000:0.0:1715208060.308454:0:3684:0:(client.c:2261:ptlrpc_check_set()) Completed RPC req@ffff8800a0219180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3684:1798523510683904:0@lo:13:osp-pre-0-0.0 00000100:00100000:3.0:1715208060.324130:0:3686:0:(client.c:1776:ptlrpc_send_new_req()) Sending RPC req@ffff8800a0219180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3686:1798523510684032:0@lo:13:osp-pre-0-1.0 00000100:00100000:3.0:1715208060.324176:0:3686:0:(events.c:359:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1715208060.324238:0:9919:0:(service.c:2096:ptlrpc_server_handle_req_in()) unwrap req x1798523510684032 00000100:00100000:2.0:1715208060.324248:0:9919:0:(service.c:2161:ptlrpc_server_handle_req_in()) got req x1798523510684032 00000100:00080000:3.0:1715208060.324284:0:7120:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 older than 1715208030 00000100:00100000:2.0:1715208060.324299:0:9919:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 1873 00000100:00100000:2.0:1715208060.324307:0:9919:0:(service.c:2335:ptlrpc_server_handle_request()) Handling RPC req@ffff88009e03d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3686:x1798523510684032:12345-0@lo:13:osp-pre-0-1.0 00002000:00000024:2.0:1715208060.324335:0:9919:0:(ofd_obd.c:743:ofd_statfs()) blocks cached 0 granted 278208 pending 0 free 3922501632 avail 3690979328 00000100:00100000:2.0:1715208060.324398:0:9919:0:(service.c:2384:ptlrpc_server_handle_request()) Handled RPC req@ffff88009e03d180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0001-mdtlov_UUID+5:3686:x1798523510684032:12345-0@lo:13:osp-pre-0-1.0 Request processed in 91us (221us total) trans 0 rc 0/0 00000100:00100000:2.0:1715208060.324407:0:9919:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 1873 00000100:00100000:3.0:1715208060.324449:0:3686:0:(client.c:2261:ptlrpc_check_set()) Completed RPC req@ffff8800a0219180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0001-mdtlov_UUID:3686:1798523510684032:0@lo:13:osp-pre-0-1.0 00010000:00010000:3.0:1715208060.354011:0:9401:0:(ldlm_lockd.c:2860:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 75 lines, 75 kept, 0 dropped, 0 bad.