00000080:00200080:0.0F:1715207786.571393:0:30561:0:(dir.c:1969:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff880136f588c8) cmd=401866d5 arg=c5e0f0 00800000:00000080:0.0:1715207786.571422:0:30561:0:(lmv_obd.c:911:lmv_iocontrol()) lustre-clilmv-ffff8800aa1c9800: cmd=401866d5 len=24 karg=ffff8800ac076380 uarg=0000000000000000 00000002:00000080:0.0:1715207786.571433:0:30561:0:(mdc_request.c:2233:mdc_iocontrol()) lustre-MDT0000-mdc-ffff8800aa1c9800: cmd=401866d5 len=24 karg=ffff8800ac076380 uarg=0000000000000000 00000100:00100000:0.0:1715207786.571488:0:30561:0:(client.c:1776:ptlrpc_send_new_req()) Sending RPC req@ffff88012aaf5f80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:cbae74da-7571-45ad-b971-0e0378bbd4e4:30561:1798523503916928:192.168.202.138@tcp:60:lhsmtool_posix.0 00000100:00100000:0.0:1715207786.571586:0:30561:0:(client.c:2542:ptlrpc_set_wait()) set ffff8800aaeeed80 going to sleep for 11 seconds 00000100:00100000:2.0F:1715207786.573219:0:30561:0:(client.c:2261:ptlrpc_check_set()) Completed RPC req@ffff88012aaf5f80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:cbae74da-7571-45ad-b971-0e0378bbd4e4:30561:1798523503916928:192.168.202.138@tcp:60:lhsmtool_posix.0 00000080:00200008:2.0:1715207786.573272:0:30561:0:(file.c:430:ll_file_release()) START file fid:[0x200000002:0x2:0x0](ffff8800b08900c8), flags 100000 00000002:00100000:2.0:1715207786.573305:0:30561:0:(mdc_request.c:927:mdc_close()) @@@ matched open req@ffff88012a614a80 x1798523503916608/t4294967442(4294967442) o101->lustre-MDT0000-mdc-ffff8800aa1c9800@192.168.202.138@tcp:12/10 lens 864/608 e 0 to 0 dl 1715207800 ref 1 fl Complete:RPQU/204/ffffffff rc 301/-1 job:'lhsmtool_posix.0' uid:0 gid:0 00000020:00100000:2.0:1715207786.573328:0:30561:0:(genops.c:2297:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8800aa1c9800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1715207786.573334:0:30561:0:(client.c:765:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8800b6325c00 x1798523503917056/t0(0) o35->lustre-MDT0000-mdc-ffff8800aa1c9800@192.168.202.138@tcp:23/10 lens 392/840 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1715207786.573349:0:30561:0:(client.c:1776:ptlrpc_send_new_req()) Sending RPC req@ffff8800b6325c00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:cbae74da-7571-45ad-b971-0e0378bbd4e4:30561:1798523503917056:192.168.202.138@tcp:35:lhsmtool_posix.0 00000100:00100000:2.0:1715207786.573401:0:30561:0:(client.c:2542:ptlrpc_set_wait()) set ffff880137391b40 going to sleep for 11 seconds 00000100:00100000:2.0:1715207786.574910:0:30561:0:(client.c:2261:ptlrpc_check_set()) Completed RPC req@ffff8800b6325c00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:cbae74da-7571-45ad-b971-0e0378bbd4e4:30561:1798523503917056:192.168.202.138@tcp:35:lhsmtool_posix.0 00000002:00100000:2.0:1715207786.574934:0:30561:0:(mdc_request.c:837:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff88012a614a80 x1798523503916608/t4294967442(4294967442) o101->lustre-MDT0000-mdc-ffff8800aa1c9800@192.168.202.138@tcp:12/10 lens 864/608 e 0 to 0 dl 1715207800 ref 1 fl Complete:RQU/204/ffffffff rc 301/-1 job:'lhsmtool_posix.0' uid:0 gid:0 00000080:00200008:2.0:1715207786.574968:0:30561:0:(file.c:430:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff880136f588c8), flags 100000 00000080:00200008:0.0:1715207786.578355:0:30561:0:(file.c:430:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff880136f588c8), flags 100000 00010000:00010000:0.0:1715207786.736187:0:4776:0:(ldlm_lockd.c:2860:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0F:1715207787.738309:0:3699:0:(ldlm_lockd.c:2860:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1715207788.636210:0:65:0:(pinger.c:253:ptlrpc_pinger_process_import()) 46af332d-2ee7-4d47-b762-82c1eb2b6bbd->lustre-MDT0000_UUID: level EVICTED/10 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1715207788.636217:0:65:0:(pinger.c:271:ptlrpc_pinger_process_import()) 46af332d-2ee7-4d47-b762-82c1eb2b6bbd->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: EVICTED) 00000100:00100000:0.0:1715207788.636218:0:1955:0:(ptlrpcd.c:417:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00080000:1.0:1715207788.636220:0:65:0:(pinger.c:253:ptlrpc_pinger_process_import()) 46af332d-2ee7-4d47-b762-82c1eb2b6bbd->lustre-OST0000_UUID: level EVICTED/10 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1715207788.636222:0:65:0:(pinger.c:271:ptlrpc_pinger_process_import()) 46af332d-2ee7-4d47-b762-82c1eb2b6bbd->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: EVICTED) 00000100:00080000:1.0:1715207788.636238:0:65:0:(pinger.c:253:ptlrpc_pinger_process_import()) cbae74da-7571-45ad-b971-0e0378bbd4e4->lustre-OST0001_UUID: level EVICTED/10 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1715207788.636241:0:65:0:(pinger.c:271:ptlrpc_pinger_process_import()) cbae74da-7571-45ad-b971-0e0378bbd4e4->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: EVICTED) 00000100:00100000:0.0:1715207788.636247:0:1955:0:(client.c:1776:ptlrpc_send_new_req()) Sending RPC req@ffff88012a614e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:ff645b43-0d7b-4a37-b9cc-6ac8f8d65c0d:1955:1798523503917120:192.168.202.138@tcp:400:kworker.0 00000100:00100000:1.0:1715207788.636263:0:1954:0:(client.c:1776:ptlrpc_send_new_req()) Sending RPC req@ffff88012a615880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:cbae74da-7571-45ad-b971-0e0378bbd4e4:1954:1798523503917248:192.168.202.138@tcp:400:kworker.0 00000100:00100000:0.0:1715207788.636347:0:1955:0:(client.c:1776:ptlrpc_send_new_req()) Sending RPC req@ffff88012a616a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:cbae74da-7571-45ad-b971-0e0378bbd4e4:1955:1798523503917184:192.168.202.138@tcp:400:kworker.0 00000100:00100000:0.0:1715207788.637783:0:1955:0:(client.c:2261:ptlrpc_check_set()) Completed RPC req@ffff88012a614e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:ff645b43-0d7b-4a37-b9cc-6ac8f8d65c0d:1955:1798523503917120:192.168.202.138@tcp:400:kworker.0 00000100:00100000:0.0:1715207788.637805:0:1955:0:(client.c:2956:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8800aa1c9800: committing for last_committed 4294967443 gen 1 00000100:00100000:1.0:1715207788.637808:0:1954:0:(client.c:2261:ptlrpc_check_set()) Completed RPC req@ffff88012a615880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:cbae74da-7571-45ad-b971-0e0378bbd4e4:1954:1798523503917248:192.168.202.138@tcp:400:kworker.0 00000100:00100000:0.0:1715207788.637810:0:1955:0:(client.c:2261:ptlrpc_check_set()) Completed RPC req@ffff88012a616a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:cbae74da-7571-45ad-b971-0e0378bbd4e4:1955:1798523503917184:192.168.202.138@tcp:400:kworker.0 00010000:00010000:1.0:1715207788.740270:0:3697:0:(ldlm_lockd.c:2860:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1715207789.742263:0:3698:0:(ldlm_lockd.c:2860:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1715207790.744255:0:4776:0:(ldlm_lockd.c:2860:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0F:1715207791.303962:0:31408:0:(debug.c:705:libcfs_debug_mark_buffer()) DEBUG MARKER: == sanity-hsm test 13: Recursively import and restore a directory ========================================================== 18:36:30 (1715207790) 00010000:00010000:1.0:1715207791.746271:0:3699:0:(ldlm_lockd.c:2860:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1715207792.748167:0:3697:0:(ldlm_lockd.c:2860:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:1.0:1715207793.385915:0:32164:0:(file.c:5701:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff880136f580c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1715207793.385929:0:32164:0:(file.c:5598:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff880136f580c8),name=/ 00000002:00010000:1.0:1715207793.385955:0:32164:0:(mdc_locks.c:1426:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1715207793.386020:0:32164:0:(ldlm_lock.c:791:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8800b0712000 lock: ffff880136bae640/0xbbeffb005ec3e36f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 32164 initiator: MDT0 00010000:00010000:1.0:1715207793.386034:0:32164:0:(ldlm_request.c:1038:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8800b0712000 lock: ffff880136bae640/0xbbeffb005ec3e36f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 32164 initiator: MDT0 00010000:00010000:1.0:1715207793.386083:0:32164:0:(ldlm_request.c:1125:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8800b0712000 lock: ffff880136bae640/0xbbeffb005ec3e36f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 32164 initiator: MDT0 00000100:00100000:1.0:1715207793.386107:0:32164:0:(client.c:2542:ptlrpc_set_wait()) set ffff88012c0e0e40 going to sleep for 0 seconds 00000100:00100000:1.0:1715207793.386122:0:32164:0:(client.c:2261:ptlrpc_check_set()) Completed RPC req@ffff88012a615880 pname:cluuid:pid:xid:nid:opc:job mkdir:46af332d-2ee7-4d47-b762-82c1eb2b6bbd:32164:1798523503917312:192.168.202.138@tcp:101:mkdir.0 00010000:00010000:1.0:1715207793.386138:0:32164:0:(ldlm_request.c:655:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff8800b0712000 lock: ffff880136bae640/0xbbeffb005ec3e36f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 32164 initiator: MDT0 00010000:00010000:1.0:1715207793.386156:0:32164:0:(ldlm_request.c:572:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff8800b0712000 lock: ffff880136bae640/0xbbeffb005ec3e36f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 32164 initiator: MDT0 00010000:00010000:1.0:1715207793.386166:0:32164:0:(ldlm_lock.c:845:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8800b0712000 lock: ffff880136bae640/0xbbeffb005ec3e36f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 32164 initiator: MDT0 00010000:00010000:1.0:1715207793.386176:0:32164:0:(ldlm_lock.c:906:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8800b0712000 lock: ffff880136bae640/0xbbeffb005ec3e36f lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 32164 initiator: MDT0 00010000:00010000:1.0:1715207793.386187:0:32164:0:(ldlm_lockd.c:1947:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8800b0712000 lock: ffff880136bae640/0xbbeffb005ec3e36f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 32164 initiator: MDT0 00010000:00010000:1.0:1715207793.386196:0:32164:0:(ldlm_lockd.c:1961:ldlm_handle_bl_callback()) Lock ffff880136bae640 already unused, calling callback (ffffffffa1044e10) 00010000:00010000:1.0:1715207793.386203:0:32164:0:(ldlm_request.c:1599:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8800b0712000 lock: ffff880136bae640/0xbbeffb005ec3e36f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 2 type: IBT flags: 0x206400000000 pid: 32164 initiator: MDT0 00010000:00010000:1.0:1715207793.386212:0:32164:0:(ldlm_request.c:1609:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8800b0712000 lock: ffff880136bae640/0xbbeffb005ec3e36f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 2 type: IBT flags: 0x206400000000 pid: 32164 initiator: MDT0 00010000:00010000:1.0:1715207793.386223:0:32164:0:(ldlm_request.c:1300:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8800b0712000 lock: ffff880136bae640/0xbbeffb005ec3e36f lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 2 type: IBT flags: 0x20e400000000 pid: 32164 initiator: MDT0 00010000:00010000:1.0:1715207793.386232:0:32164:0:(ldlm_request.c:1316:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1715207793.386239:0:32164:0:(ldlm_lockd.c:1971:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8800b0712000 lock: ffff880136bae640/0xbbeffb005ec3e36f lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 32164 initiator: MDT0 00010000:00010000:1.0:1715207793.386248:0:32164:0:(ldlm_lock.c:215:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8800b0712000 lock: ffff880136bae640/0xbbeffb005ec3e36f lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 32164 initiator: MDT0 00000080:00020000:1.0:1715207793.386278:0:32164:0:(file.c:5575:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108 00000100:00080000:1.0:1715207793.644135:0:12761:0:(pinger.c:253:ptlrpc_pinger_process_import()) 46af332d-2ee7-4d47-b762-82c1eb2b6bbd->lustre-MDT0000_UUID: level EVICTED/10 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1715207793.644140:0:12761:0:(pinger.c:271:ptlrpc_pinger_process_import()) 46af332d-2ee7-4d47-b762-82c1eb2b6bbd->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: EVICTED) 00000100:00080000:1.0:1715207793.644147:0:12761:0:(pinger.c:253:ptlrpc_pinger_process_import()) 46af332d-2ee7-4d47-b762-82c1eb2b6bbd->lustre-OST0000_UUID: level EVICTED/10 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1715207793.644149:0:12761:0:(pinger.c:271:ptlrpc_pinger_process_import()) 46af332d-2ee7-4d47-b762-82c1eb2b6bbd->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: EVICTED) 00000100:00080000:1.0:1715207793.644160:0:12761:0:(pinger.c:253:ptlrpc_pinger_process_import()) cbae74da-7571-45ad-b971-0e0378bbd4e4->lustre-OST0001_UUID: level EVICTED/10 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1715207793.644162:0:12761:0:(pinger.c:271:ptlrpc_pinger_process_import()) cbae74da-7571-45ad-b971-0e0378bbd4e4->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: EVICTED) 00000100:00100000:1.0:1715207793.644182:0:1954:0:(client.c:1776:ptlrpc_send_new_req()) Sending RPC req@ffff88012a617b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:cbae74da-7571-45ad-b971-0e0378bbd4e4:1954:1798523503917440:192.168.202.138@tcp:400:kworker.0 00000100:00100000:0.0:1715207793.644191:0:1955:0:(client.c:1776:ptlrpc_send_new_req()) Sending RPC req@ffff88012a615880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:ff645b43-0d7b-4a37-b9cc-6ac8f8d65c0d:1955:1798523503917376:192.168.202.138@tcp:400:kworker.0 00000100:00100000:0.0:1715207793.644235:0:1955:0:(client.c:1776:ptlrpc_send_new_req()) Sending RPC req@ffff88012a616d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:cbae74da-7571-45ad-b971-0e0378bbd4e4:1955:1798523503917504:192.168.202.138@tcp:400:kworker.0 00000100:00100000:1.0:1715207793.645238:0:1955:0:(client.c:2261:ptlrpc_check_set()) Completed RPC req@ffff88012a615880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:ff645b43-0d7b-4a37-b9cc-6ac8f8d65c0d:1955:1798523503917376:192.168.202.138@tcp:400:kworker.0 00000100:00100000:1.0:1715207793.645247:0:1955:0:(client.c:2261:ptlrpc_check_set()) Completed RPC req@ffff88012a616d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:cbae74da-7571-45ad-b971-0e0378bbd4e4:1955:1798523503917504:192.168.202.138@tcp:400:kworker.0 00000100:00100000:0.0:1715207793.645251:0:1954:0:(client.c:2261:ptlrpc_check_set()) Completed RPC req@ffff88012a617b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:cbae74da-7571-45ad-b971-0e0378bbd4e4:1954:1798523503917440:192.168.202.138@tcp:400:kworker.0 00010000:00010000:0.0:1715207793.750312:0:3698:0:(ldlm_lockd.c:2860:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1715207794.268201:0:32297:0:(debug.c:705:libcfs_debug_mark_buffer()) DEBUG MARKER: sanity-hsm test_13: @@@@@@ FAIL: cannot create directory '/mnt/lustre' 00010000:00010000:1.0:1715207794.752228:0:4776:0:(ldlm_lockd.c:2860:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 74 lines, 74 kept, 0 dropped, 0 bad.