-----============= acceptance-small: sanity-hsm ============----- Mon Jun 2 08:28:22 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg226-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-hsm.*ex: No such file or directory excepting tests: === sanity-hsm: start setup 08:28:31 (1748867311) === Starting client oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre2 Started clients oleg226-client.virtnet: 192.168.202.126@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg226-client.virtnet: executing check_config_client /mnt/lustre oleg226-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg226-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800aa400800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8800b6d11000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800aa400800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6d11000.idle_timeout=debug disable quota as required oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-hsm: finish setup 08:28:47 (1748867327) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7959] nobody:x:99: Killing existing copytools on oleg226-client.virtnet pkill: pidfile not valid Try `pkill --help' for more information. Set HSM on and start Set sanity-hsm HSM policy == sanity-hsm test 1A: lfs hsm flags root/non-root access ========================================================== 08:29:11 (1748867351) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_set] [--norelease] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_clear] [--norelease] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_set] [--exists] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] lfs hsm_set: change hsm flags for '/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm' failed: Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] PASS 1A (3s) == sanity-hsm test 1a: mmap & cat a HSM released file ==== 08:29:14 (1748867354) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.155262 s, 6.8 MB/s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_1a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Verifying released state: PASS 1a (7s) == sanity-hsm test 1b: Archive, Release and Restore composite file ========================================================== 08:29:21 (1748867361) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.02984 s, 35.1 MB/s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_1b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' archive /mnt/lustre/d1b.sanity-hsm/f1b.sanity-hsm Waiting 200s for 'SUCCEED' release /mnt/lustre/d1b.sanity-hsm/f1b.sanity-hsm verify released state: pass restore /mnt/lustre/d1b.sanity-hsm/f1b.sanity-hsm Waiting 200s for 'SUCCEED' verify restored state: pass PASS 1b (6s) == sanity-hsm test 1c: Check setting archive-id in lfs hsm_set ========================================================== 08:29:27 (1748867367) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d1c.sanity-hsm/f1c.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1c.sanity-hsm/f1c.sanity-hsm] verifying archive number is 32 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1c.sanity-hsm/f1c.sanity-hsm] verifying archive number is still 32 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1c.sanity-hsm/f1c.sanity-hsm] verifying archive number is 33 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1c.sanity-hsm/f1c.sanity-hsm] verifying archive number is 16 PASS 1c (3s) == sanity-hsm test 1d: Archive, Release and Restore DoM file ========================================================== 08:29:30 (1748867370) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0250879 s, 41.8 MB/s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_1d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' archive /mnt/lustre/d1d.sanity-hsm/f1d.sanity-hsm Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' release /mnt/lustre/d1d.sanity-hsm/f1d.sanity-hsm verify released state: pass restore /mnt/lustre/d1d.sanity-hsm/f1d.sanity-hsm Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' verify restored state: pass PASS 1d (7s) == sanity-hsm test 1e: Archive, Release and Restore SEL file ========================================================== 08:29:37 (1748867377) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0247841 s, 42.3 MB/s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_1e/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' archive /mnt/lustre/d1e.sanity-hsm/f1e.sanity-hsm Waiting 200s for 'SUCCEED' release /mnt/lustre/d1e.sanity-hsm/f1e.sanity-hsm verify released state: pass restore /mnt/lustre/d1e.sanity-hsm/f1e.sanity-hsm verify restored state: pass PASS 1e (6s) == sanity-hsm test 1f: DoM file release after restore ==== 08:29:43 (1748867383) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0593561 s, 17.7 MB/s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_1f/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' archive /mnt/lustre/d1f.sanity-hsm/f1f.sanity-hsm Waiting 200s for 'SUCCEED' release /mnt/lustre/d1f.sanity-hsm/f1f.sanity-hsm verify released state: pass restore /mnt/lustre/d1f.sanity-hsm/f1f.sanity-hsm Waiting 200s for 'SUCCEED' verify restored state: pass release again /mnt/lustre/d1f.sanity-hsm/f1f.sanity-hsm /mnt/lustre/d1f.sanity-hsm/f1f.sanity-hsm: (0x0000000d) released exists archived, archive_id:1 verify released state: pass PASS 1f (6s) == sanity-hsm test 2: Check file dirtyness when doing setattr ========================================================== 08:29:49 (1748867389) 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00202761 s, 0.0 kB/s 0x200000403:0x8:0x0 PASS 2 (3s) == sanity-hsm test 3: Check file dirtyness when opening for write ========================================================== 08:29:52 (1748867392) Succeed in opening file "/mnt/lustre/d3.sanity-hsm/f3.sanity-hsm"(flags=O_WRONLY) 3+0 records in 3+0 records out 3+0 records in 3+0 records out PASS 3 (3s) == sanity-hsm test 4: Useless cancel must not be registered ========================================================== 08:29:55 (1748867395) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0884906 s, 11.8 MB/s PASS 4 (3s) == sanity-hsm test 8: Test default archive number ======== 08:29:58 (1748867398) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_8/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' PASS 8 (5s) == sanity-hsm test 9A: Use of explicit archive number, with dedicated copytool ========================================================== 08:30:03 (1748867403) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_9A/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 3 "/mnt/lustre2"' oleg226-client.virtnet: executing pgrep --pidfile=/var/run/lhsmtool_posix.pid --list-full hsmtool oleg226-client.virtnet: executing get_client_uuid /mnt/lustre2 found agent uuid=c8af5ef9-f4c9-4606-93aa-65a65d478e84 archive_id=3 requests=[current:0 ok:0 errors:0] on mds1 found agent uuid=c8af5ef9-f4c9-4606-93aa-65a65d478e84 archive_id=3 requests=[current:0 ok:0 errors:0] on mds2 PASS 9A (12s) == sanity-hsm test 9a: Multiple remote agents ============ 08:30:15 (1748867415) SKIP: sanity-hsm test_9a Need 3 or more clients, have 1 SKIP 9a (2s) == sanity-hsm test 10a: Archive a file =================== 08:30:17 (1748867417) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_10a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Verifying content Verifying hsm state Verifying archive number is 2 PASS 10a (5s) == sanity-hsm test 10b: Archive of non dirty file must work without doing request ========================================================== 08:30:22 (1748867422) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_10b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' PASS 10b (3s) == sanity-hsm test 10c: Check forbidden archive ========== 08:30:25 (1748867425) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_10c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Cannot send HSM request (use of /mnt/lustre/d10c.sanity-hsm/f10c.sanity-hsm): Operation not permitted PASS 10c (3s) == sanity-hsm test 10d: Archive a file on the default archive id ========================================================== 08:30:28 (1748867428) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_10d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' PASS 10d (4s) == sanity-hsm test 11a: Import a file ==================== 08:30:32 (1748867432) lhsmtool_posix: 1748867433.153503 lhsmtool_posix[22939]: action=1 src=d11a.sanity-hsm/f11a.sanity-hsm dst=/mnt/lustre/d11a.sanity-hsm/f11a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1748867433.160288 lhsmtool_posix[22939]: importing '/mnt/lustre/d11a.sanity-hsm/f11a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_11a//d11a.sanity-hsm/f11a.sanity-hsm' lhsmtool_posix: 1748867433.183780 lhsmtool_posix[22939]: imported '/mnt/lustre/d11a.sanity-hsm/f11a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_11a//0424/0x200000402:0x26:0x0'=='/tmp/arc1/sanity-hsm.test_11a//d11a.sanity-hsm/f11a.sanity-hsm' lhsmtool_posix: 1748867433.183942 lhsmtool_posix[22939]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Verifying released state: Verifying imported size 159=159 Verifying released pattern: released Verifying new fid 0x200000402:0x26:0x0 in archive PASS 11a (3s) == sanity-hsm test 11b: Import a deleted file using its FID ========================================================== 08:30:35 (1748867435) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_11b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' lhsmtool_posix: 1748867438.338602 lhsmtool_posix[23697]: action=1 src=0x200000403:0x11:0x0 dst=/mnt/lustre/d11b.sanity-hsm/f11b.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1748867438.348497 lhsmtool_posix[23697]: Resolving [0x200000403:0x11:0x0] to /tmp/arc1/sanity-hsm.test_11b//0412/0x200000403:0x11:0x0 lhsmtool_posix: 1748867438.348588 lhsmtool_posix[23697]: importing '/mnt/lustre/d11b.sanity-hsm/f11b.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_11b//0412/0x200000403:0x11:0x0' lhsmtool_posix: 1748867438.375489 lhsmtool_posix[23697]: imported '/mnt/lustre/d11b.sanity-hsm/f11b.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_11b//042a/0x200000402:0x28:0x0'=='/tmp/arc1/sanity-hsm.test_11b//0412/0x200000403:0x11:0x0' lhsmtool_posix: 1748867438.375571 lhsmtool_posix[23697]: process finished, errs: 0 major, 0 minor, rc=0 (Success) /mnt/lustre/d11b.sanity-hsm/f11b.sanity-hsm: OK PASS 11b (6s) == sanity-hsm test 11c: Import a file to a directory with a pool ========================================================== 08:30:41 (1748867441) Creating new pool test_11c oleg226-server: Pool lustre.test_11c created Adding targets to pool oleg226-server: OST lustre-OST0001_UUID added to pool lustre.test_11c lhsmtool_posix: 1748867446.321747 lhsmtool_posix[24473]: action=1 src=d11c.sanity-hsm/f11c.sanity-hsm dst=/mnt/lustre/d11c.sanity-hsm/f11c.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1748867446.325712 lhsmtool_posix[24473]: importing '/mnt/lustre/d11c.sanity-hsm/f11c.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_11c//d11c.sanity-hsm/f11c.sanity-hsm' lhsmtool_posix: 1748867446.341921 lhsmtool_posix[24473]: imported '/mnt/lustre/d11c.sanity-hsm/f11c.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_11c//0428/0x200000402:0x2a:0x0'=='/tmp/arc1/sanity-hsm.test_11c//d11c.sanity-hsm/f11c.sanity-hsm' lhsmtool_posix: 1748867446.341997 lhsmtool_posix[24473]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Destroy the created pools: test_11c lustre.test_11c oleg226-server: OST lustre-OST0001_UUID removed from pool lustre.test_11c oleg226-server: Pool lustre.test_11c destroyed PASS 11c (12s) == sanity-hsm test 12a: Restore an imported file explicitly ========================================================== 08:30:53 (1748867453) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1748867454.339208 lhsmtool_posix[25271]: action=1 src=d12a.sanity-hsm/f12a.sanity-hsm dst=/mnt/lustre/d12a.sanity-hsm/f12a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1748867454.342204 lhsmtool_posix[25271]: importing '/mnt/lustre/d12a.sanity-hsm/f12a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12a//d12a.sanity-hsm/f12a.sanity-hsm' lhsmtool_posix: 1748867454.368833 lhsmtool_posix[25271]: imported '/mnt/lustre/d12a.sanity-hsm/f12a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12a//042f/0x200000402:0x2d:0x0'=='/tmp/arc1/sanity-hsm.test_12a//d12a.sanity-hsm/f12a.sanity-hsm' lhsmtool_posix: 1748867454.369788 lhsmtool_posix[25271]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Verifying released state: Waiting 200s for 'SUCCEED' Verifying file state: PASS 12a (4s) == sanity-hsm test 12b: Restore an imported file implicitly ========================================================== 08:30:58 (1748867458) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1748867459.039590 lhsmtool_posix[26044]: action=1 src=d12b.sanity-hsm/f12b.sanity-hsm dst=/mnt/lustre/d12b.sanity-hsm/f12b.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1748867459.049081 lhsmtool_posix[26044]: importing '/mnt/lustre/d12b.sanity-hsm/f12b.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12b//d12b.sanity-hsm/f12b.sanity-hsm' lhsmtool_posix: 1748867459.069193 lhsmtool_posix[26044]: imported '/mnt/lustre/d12b.sanity-hsm/f12b.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12b//042d/0x200000402:0x2f:0x0'=='/tmp/arc1/sanity-hsm.test_12b//d12b.sanity-hsm/f12b.sanity-hsm' lhsmtool_posix: 1748867459.069255 lhsmtool_posix[26044]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Verifying released state: Verifying file state after restore: PASS 12b (3s) == sanity-hsm test 12c: Restore a file with stripe of 2 == 08:31:01 (1748867461) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.0630801 s, 83.1 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' /mnt/lustre/d12c.sanity-hsm/f12c.sanity-hsm: OK PASS 12c (5s) == sanity-hsm test 12d: Restore of a non archived, non released file must work ========================================================== 08:31:06 (1748867466) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' PASS 12d (4s) == sanity-hsm test 12e: Check forbidden restore ========== 08:31:10 (1748867470) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12e/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' /mnt/lustre/d12e.sanity-hsm/f12e.sanity-hsm: (0x0000000b) exists dirty archived, archive_id:1 Cannot send HSM request (use of /mnt/lustre/d12e.sanity-hsm/f12e.sanity-hsm): Operation not permitted PASS 12e (3s) == sanity-hsm test 12f: Restore a released file explicitly ========================================================== 08:31:13 (1748867473) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12f/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Verifying file state: PASS 12f (5s) == sanity-hsm test 12g: Restore a released file implicitly ========================================================== 08:31:18 (1748867478) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12g/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' PASS 12g (4s) == sanity-hsm test 12h: Restore a released file implicitly from a second node ========================================================== 08:31:22 (1748867482) SKIP: sanity-hsm test_12h Need 2 or more clients, have 1 SKIP 12h (2s) == sanity-hsm test 12m: Archive/release/implicit restore ========================================================== 08:31:24 (1748867484) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12m/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' PASS 12m (4s) == sanity-hsm test 12n: Import/implicit restore/release == 08:31:28 (1748867488) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12n/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1748867489.445748 lhsmtool_posix[31317]: action=1 src=d12n.sanity-hsm/f12n.sanity-hsm dst=/mnt/lustre/d12n.sanity-hsm/f12n.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1748867489.451515 lhsmtool_posix[31317]: importing '/mnt/lustre/d12n.sanity-hsm/f12n.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12n//d12n.sanity-hsm/f12n.sanity-hsm' lhsmtool_posix: 1748867489.472953 lhsmtool_posix[31317]: imported '/mnt/lustre/d12n.sanity-hsm/f12n.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12n//043f/0x200000402:0x3d:0x0'=='/tmp/arc1/sanity-hsm.test_12n//d12n.sanity-hsm/f12n.sanity-hsm' lhsmtool_posix: 1748867489.473023 lhsmtool_posix[31317]: process finished, errs: 0 major, 0 minor, rc=0 (Success) PASS 12n (3s) == sanity-hsm test 12o: Layout-swap failure during Restore leaves file released ========================================================== 08:31:31 (1748867491) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12o/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' fail_loc=0x152 mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0001.hsm.policy=+NRA diff: /mnt/lustre/d12o.sanity-hsm/f12o.sanity-hsm: No data available mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA fail_loc=0 mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge Waiting 200s for 'SUCCEED' PASS 12o (22s) == sanity-hsm test 12p: implicit restore of a file on copytool mount point ========================================================== 08:31:53 (1748867513) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12p/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' PASS 12p (4s) == sanity-hsm test 12q: file attributes are refreshed after restore ========================================================== 08:31:57 (1748867517) Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre3 Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12q/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre3"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0699239 s, 15.0 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Killing existing copytools on oleg226-client.virtnet copytools stopped in 0s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12q/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre3"' Stopping client oleg226-client.virtnet /mnt/lustre3 (opts:) PASS 12q (11s) == sanity-hsm test 12r: lseek restores released file ===== 08:32:08 (1748867528) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12r/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' PASS 12r (5s) == sanity-hsm test 12s: race between restore requests ==== 08:32:13 (1748867533) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12s/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' fail_loc=0x8000018b PASS 12s (9s) == sanity-hsm test 12t: Multiple parallel reads for a HSM imported file ========================================================== 08:32:22 (1748867542) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12t/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1748867544.253495 lhsmtool_posix[3437]: action=1 src=0x200000403:0x2a:0x0 dst=/mnt/lustre/d12t.sanity-hsm/f12t.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1748867544.257337 lhsmtool_posix[3437]: Resolving [0x200000403:0x2a:0x0] to /tmp/arc1/sanity-hsm.test_12t//0429/0x200000403:0x2a:0x0 lhsmtool_posix: 1748867544.257414 lhsmtool_posix[3437]: importing '/mnt/lustre/d12t.sanity-hsm/f12t.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12t//0429/0x200000403:0x2a:0x0' lhsmtool_posix: 1748867544.283582 lhsmtool_posix[3437]: imported '/mnt/lustre/d12t.sanity-hsm/f12t.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12t//0449/0x200000402:0x4b:0x0'=='/tmp/arc1/sanity-hsm.test_12t//0429/0x200000403:0x2a:0x0' lhsmtool_posix: 1748867544.283658 lhsmtool_posix[3437]: process finished, errs: 0 major, 0 minor, rc=0 (Success) PASS 12t (4s) == sanity-hsm test 12u: Multiple reads on multiple HSM imported files in parallel ========================================================== 08:32:26 (1748867546) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12u/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1748867551.835150 lhsmtool_posix[4370]: action=1 src=0x200000403:0x2c:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.0 mount_point=/mnt/lustre2 lhsmtool_posix: 1748867551.839275 lhsmtool_posix[4370]: Resolving [0x200000403:0x2c:0x0] to /tmp/arc1/sanity-hsm.test_12u//042f/0x200000403:0x2c:0x0 lhsmtool_posix: 1748867551.839567 lhsmtool_posix[4370]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.0' from '/tmp/arc1/sanity-hsm.test_12u//042f/0x200000403:0x2c:0x0' lhsmtool_posix: 1748867551.870427 lhsmtool_posix[4370]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.0' from '/tmp/arc1/sanity-hsm.test_12u//044f/0x200000402:0x4d:0x0'=='/tmp/arc1/sanity-hsm.test_12u//042f/0x200000403:0x2c:0x0' lhsmtool_posix: 1748867551.870530 lhsmtool_posix[4370]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1748867551.975559 lhsmtool_posix[4399]: action=1 src=0x200000403:0x2d:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.1 mount_point=/mnt/lustre2 lhsmtool_posix: 1748867551.979413 lhsmtool_posix[4399]: Resolving [0x200000403:0x2d:0x0] to /tmp/arc1/sanity-hsm.test_12u//042e/0x200000403:0x2d:0x0 lhsmtool_posix: 1748867551.979488 lhsmtool_posix[4399]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.1' from '/tmp/arc1/sanity-hsm.test_12u//042e/0x200000403:0x2d:0x0' lhsmtool_posix: 1748867552.001654 lhsmtool_posix[4399]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.1' from '/tmp/arc1/sanity-hsm.test_12u//044c/0x200000402:0x4e:0x0'=='/tmp/arc1/sanity-hsm.test_12u//042e/0x200000403:0x2d:0x0' lhsmtool_posix: 1748867552.001728 lhsmtool_posix[4399]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1748867552.088686 lhsmtool_posix[4428]: action=1 src=0x200000403:0x2e:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.2 mount_point=/mnt/lustre2 lhsmtool_posix: 1748867552.092772 lhsmtool_posix[4428]: Resolving [0x200000403:0x2e:0x0] to /tmp/arc1/sanity-hsm.test_12u//042d/0x200000403:0x2e:0x0 lhsmtool_posix: 1748867552.092825 lhsmtool_posix[4428]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.2' from '/tmp/arc1/sanity-hsm.test_12u//042d/0x200000403:0x2e:0x0' lhsmtool_posix: 1748867552.114099 lhsmtool_posix[4428]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.2' from '/tmp/arc1/sanity-hsm.test_12u//044d/0x200000402:0x4f:0x0'=='/tmp/arc1/sanity-hsm.test_12u//042d/0x200000403:0x2e:0x0' lhsmtool_posix: 1748867552.114412 lhsmtool_posix[4428]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1748867552.222446 lhsmtool_posix[4457]: action=1 src=0x200000403:0x2f:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.3 mount_point=/mnt/lustre2 lhsmtool_posix: 1748867552.227526 lhsmtool_posix[4457]: Resolving [0x200000403:0x2f:0x0] to /tmp/arc1/sanity-hsm.test_12u//042c/0x200000403:0x2f:0x0 lhsmtool_posix: 1748867552.228610 lhsmtool_posix[4457]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.3' from '/tmp/arc1/sanity-hsm.test_12u//042c/0x200000403:0x2f:0x0' lhsmtool_posix: 1748867552.268915 lhsmtool_posix[4457]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.3' from '/tmp/arc1/sanity-hsm.test_12u//0452/0x200000402:0x50:0x0'=='/tmp/arc1/sanity-hsm.test_12u//042c/0x200000403:0x2f:0x0' lhsmtool_posix: 1748867552.268987 lhsmtool_posix[4457]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1748867552.367805 lhsmtool_posix[4486]: action=1 src=0x200000403:0x30:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.4 mount_point=/mnt/lustre2 lhsmtool_posix: 1748867552.372063 lhsmtool_posix[4486]: Resolving [0x200000403:0x30:0x0] to /tmp/arc1/sanity-hsm.test_12u//0433/0x200000403:0x30:0x0 lhsmtool_posix: 1748867552.372150 lhsmtool_posix[4486]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.4' from '/tmp/arc1/sanity-hsm.test_12u//0433/0x200000403:0x30:0x0' lhsmtool_posix: 1748867552.391270 lhsmtool_posix[4486]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.4' from '/tmp/arc1/sanity-hsm.test_12u//0453/0x200000402:0x51:0x0'=='/tmp/arc1/sanity-hsm.test_12u//0433/0x200000403:0x30:0x0' lhsmtool_posix: 1748867552.391338 lhsmtool_posix[4486]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1748867552.492441 lhsmtool_posix[4515]: action=1 src=0x200000403:0x31:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.5 mount_point=/mnt/lustre2 lhsmtool_posix: 1748867552.495228 lhsmtool_posix[4515]: Resolving [0x200000403:0x31:0x0] to /tmp/arc1/sanity-hsm.test_12u//0432/0x200000403:0x31:0x0 lhsmtool_posix: 1748867552.495275 lhsmtool_posix[4515]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.5' from '/tmp/arc1/sanity-hsm.test_12u//0432/0x200000403:0x31:0x0' lhsmtool_posix: 1748867552.517101 lhsmtool_posix[4515]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.5' from '/tmp/arc1/sanity-hsm.test_12u//0450/0x200000402:0x52:0x0'=='/tmp/arc1/sanity-hsm.test_12u//0432/0x200000403:0x31:0x0' lhsmtool_posix: 1748867552.517189 lhsmtool_posix[4515]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1748867552.636273 lhsmtool_posix[4544]: action=1 src=0x200000403:0x32:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.6 mount_point=/mnt/lustre2 lhsmtool_posix: 1748867552.641788 lhsmtool_posix[4544]: Resolving [0x200000403:0x32:0x0] to /tmp/arc1/sanity-hsm.test_12u//0431/0x200000403:0x32:0x0 lhsmtool_posix: 1748867552.642011 lhsmtool_posix[4544]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.6' from '/tmp/arc1/sanity-hsm.test_12u//0431/0x200000403:0x32:0x0' lhsmtool_posix: 1748867552.656252 lhsmtool_posix[4544]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.6' from '/tmp/arc1/sanity-hsm.test_12u//0451/0x200000402:0x53:0x0'=='/tmp/arc1/sanity-hsm.test_12u//0431/0x200000403:0x32:0x0' lhsmtool_posix: 1748867552.656317 lhsmtool_posix[4544]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1748867552.755975 lhsmtool_posix[4573]: action=1 src=0x200000403:0x33:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.7 mount_point=/mnt/lustre2 lhsmtool_posix: 1748867552.758959 lhsmtool_posix[4573]: Resolving [0x200000403:0x33:0x0] to /tmp/arc1/sanity-hsm.test_12u//0430/0x200000403:0x33:0x0 lhsmtool_posix: 1748867552.759584 lhsmtool_posix[4573]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.7' from '/tmp/arc1/sanity-hsm.test_12u//0430/0x200000403:0x33:0x0' lhsmtool_posix: 1748867552.777744 lhsmtool_posix[4573]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.7' from '/tmp/arc1/sanity-hsm.test_12u//0456/0x200000402:0x54:0x0'=='/tmp/arc1/sanity-hsm.test_12u//0430/0x200000403:0x33:0x0' lhsmtool_posix: 1748867552.777807 lhsmtool_posix[4573]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1748867552.897708 lhsmtool_posix[4603]: action=1 src=0x200000403:0x34:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.8 mount_point=/mnt/lustre2 lhsmtool_posix: 1748867552.901255 lhsmtool_posix[4603]: Resolving [0x200000403:0x34:0x0] to /tmp/arc1/sanity-hsm.test_12u//0437/0x200000403:0x34:0x0 lhsmtool_posix: 1748867552.901317 lhsmtool_posix[4603]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.8' from '/tmp/arc1/sanity-hsm.test_12u//0437/0x200000403:0x34:0x0' lhsmtool_posix: 1748867552.918832 lhsmtool_posix[4603]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.8' from '/tmp/arc1/sanity-hsm.test_12u//0457/0x200000402:0x55:0x0'=='/tmp/arc1/sanity-hsm.test_12u//0437/0x200000403:0x34:0x0' lhsmtool_posix: 1748867552.918876 lhsmtool_posix[4603]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1748867553.005622 lhsmtool_posix[4632]: action=1 src=0x200000403:0x35:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.9 mount_point=/mnt/lustre2 lhsmtool_posix: 1748867553.009368 lhsmtool_posix[4632]: Resolving [0x200000403:0x35:0x0] to /tmp/arc1/sanity-hsm.test_12u//0436/0x200000403:0x35:0x0 lhsmtool_posix: 1748867553.009451 lhsmtool_posix[4632]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.9' from '/tmp/arc1/sanity-hsm.test_12u//0436/0x200000403:0x35:0x0' lhsmtool_posix: 1748867553.024678 lhsmtool_posix[4632]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.9' from '/tmp/arc1/sanity-hsm.test_12u//0454/0x200000402:0x56:0x0'=='/tmp/arc1/sanity-hsm.test_12u//0436/0x200000403:0x35:0x0' lhsmtool_posix: 1748867553.025029 lhsmtool_posix[4632]: process finished, errs: 0 major, 0 minor, rc=0 (Success) PASS 12u (9s) == sanity-hsm test 13: Recursively import and restore a directory ========================================================== 08:32:36 (1748867556) lhsmtool_posix: 1748867558.848031 lhsmtool_posix[6448]: action=1 src=subdir dst=/mnt/lustre/d13.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1748867558.893563 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.10' lhsmtool_posix: 1748867558.920413 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.10' from '/tmp/arc1/sanity-hsm.test_13//045b/0x200000402:0x59:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.10' lhsmtool_posix: 1748867558.921512 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.9' lhsmtool_posix: 1748867558.939238 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.9' from '/tmp/arc1/sanity-hsm.test_13//0458/0x200000402:0x5a:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.9' lhsmtool_posix: 1748867558.939386 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.8' lhsmtool_posix: 1748867558.952879 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.8' from '/tmp/arc1/sanity-hsm.test_13//0459/0x200000402:0x5b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.8' lhsmtool_posix: 1748867558.953019 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.7' lhsmtool_posix: 1748867558.967556 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.7' from '/tmp/arc1/sanity-hsm.test_13//045e/0x200000402:0x5c:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.7' lhsmtool_posix: 1748867558.967669 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.6' lhsmtool_posix: 1748867558.978414 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.6' from '/tmp/arc1/sanity-hsm.test_13//045f/0x200000402:0x5d:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.6' lhsmtool_posix: 1748867558.978553 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.5' lhsmtool_posix: 1748867558.997963 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.5' from '/tmp/arc1/sanity-hsm.test_13//045c/0x200000402:0x5e:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.5' lhsmtool_posix: 1748867558.998095 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.4' lhsmtool_posix: 1748867559.016017 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.4' from '/tmp/arc1/sanity-hsm.test_13//045d/0x200000402:0x5f:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.4' lhsmtool_posix: 1748867559.016796 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.3' lhsmtool_posix: 1748867559.036086 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.3' from '/tmp/arc1/sanity-hsm.test_13//0462/0x200000402:0x60:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.3' lhsmtool_posix: 1748867559.036543 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.2' lhsmtool_posix: 1748867559.058934 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.2' from '/tmp/arc1/sanity-hsm.test_13//0463/0x200000402:0x61:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.2' lhsmtool_posix: 1748867559.059094 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.1' lhsmtool_posix: 1748867559.074031 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.1' from '/tmp/arc1/sanity-hsm.test_13//0460/0x200000402:0x62:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.1' lhsmtool_posix: 1748867559.076915 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.10' lhsmtool_posix: 1748867559.097761 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.10' from '/tmp/arc1/sanity-hsm.test_13//0408/0x240000402:0xa:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.10' lhsmtool_posix: 1748867559.098081 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.9' lhsmtool_posix: 1748867559.116622 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.9' from '/tmp/arc1/sanity-hsm.test_13//0409/0x240000402:0xb:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.9' lhsmtool_posix: 1748867559.116761 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.8' lhsmtool_posix: 1748867559.138819 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.8' from '/tmp/arc1/sanity-hsm.test_13//040e/0x240000402:0xc:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.8' lhsmtool_posix: 1748867559.139585 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.7' lhsmtool_posix: 1748867559.154404 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.7' from '/tmp/arc1/sanity-hsm.test_13//040f/0x240000402:0xd:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.7' lhsmtool_posix: 1748867559.154567 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.6' lhsmtool_posix: 1748867559.170886 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.6' from '/tmp/arc1/sanity-hsm.test_13//040c/0x240000402:0xe:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.6' lhsmtool_posix: 1748867559.171051 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.5' lhsmtool_posix: 1748867559.191037 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.5' from '/tmp/arc1/sanity-hsm.test_13//040d/0x240000402:0xf:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.5' lhsmtool_posix: 1748867559.191328 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.4' lhsmtool_posix: 1748867559.213346 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.4' from '/tmp/arc1/sanity-hsm.test_13//0412/0x240000402:0x10:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.4' lhsmtool_posix: 1748867559.213499 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.3' lhsmtool_posix: 1748867559.232510 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.3' from '/tmp/arc1/sanity-hsm.test_13//0413/0x240000402:0x11:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.3' lhsmtool_posix: 1748867559.232731 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.2' lhsmtool_posix: 1748867559.256409 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.2' from '/tmp/arc1/sanity-hsm.test_13//0410/0x240000402:0x12:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.2' lhsmtool_posix: 1748867559.257005 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.1' lhsmtool_posix: 1748867559.279776 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.1' from '/tmp/arc1/sanity-hsm.test_13//0411/0x240000402:0x13:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.1' lhsmtool_posix: 1748867559.288102 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.10' lhsmtool_posix: 1748867559.317536 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.10' from '/tmp/arc1/sanity-hsm.test_13//0466/0x200000402:0x64:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.10' lhsmtool_posix: 1748867559.317698 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.9' lhsmtool_posix: 1748867559.338333 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.9' from '/tmp/arc1/sanity-hsm.test_13//0467/0x200000402:0x65:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.9' lhsmtool_posix: 1748867559.338485 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.8' lhsmtool_posix: 1748867559.353824 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.8' from '/tmp/arc1/sanity-hsm.test_13//0464/0x200000402:0x66:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.8' lhsmtool_posix: 1748867559.354009 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.7' lhsmtool_posix: 1748867559.372984 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.7' from '/tmp/arc1/sanity-hsm.test_13//0465/0x200000402:0x67:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.7' lhsmtool_posix: 1748867559.373171 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.6' lhsmtool_posix: 1748867559.388695 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.6' from '/tmp/arc1/sanity-hsm.test_13//046a/0x200000402:0x68:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.6' lhsmtool_posix: 1748867559.388880 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.5' lhsmtool_posix: 1748867559.407779 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.5' from '/tmp/arc1/sanity-hsm.test_13//046b/0x200000402:0x69:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.5' lhsmtool_posix: 1748867559.407944 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.4' lhsmtool_posix: 1748867559.423724 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.4' from '/tmp/arc1/sanity-hsm.test_13//0468/0x200000402:0x6a:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.4' lhsmtool_posix: 1748867559.423877 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.3' lhsmtool_posix: 1748867559.443147 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.3' from '/tmp/arc1/sanity-hsm.test_13//0469/0x200000402:0x6b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.3' lhsmtool_posix: 1748867559.443512 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.2' lhsmtool_posix: 1748867559.462536 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.2' from '/tmp/arc1/sanity-hsm.test_13//046e/0x200000402:0x6c:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.2' lhsmtool_posix: 1748867559.462832 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.1' lhsmtool_posix: 1748867559.482025 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.1' from '/tmp/arc1/sanity-hsm.test_13//046f/0x200000402:0x6d:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.1' lhsmtool_posix: 1748867559.486382 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.10' lhsmtool_posix: 1748867559.507556 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.10' from '/tmp/arc1/sanity-hsm.test_13//0417/0x240000402:0x15:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.10' lhsmtool_posix: 1748867559.507716 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.9' lhsmtool_posix: 1748867559.529528 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.9' from '/tmp/arc1/sanity-hsm.test_13//0414/0x240000402:0x16:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.9' lhsmtool_posix: 1748867559.529691 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.8' lhsmtool_posix: 1748867559.552897 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.8' from '/tmp/arc1/sanity-hsm.test_13//0415/0x240000402:0x17:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.8' lhsmtool_posix: 1748867559.553057 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.7' lhsmtool_posix: 1748867559.567049 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.7' from '/tmp/arc1/sanity-hsm.test_13//041a/0x240000402:0x18:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.7' lhsmtool_posix: 1748867559.567340 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.6' lhsmtool_posix: 1748867559.586722 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.6' from '/tmp/arc1/sanity-hsm.test_13//041b/0x240000402:0x19:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.6' lhsmtool_posix: 1748867559.586992 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.5' lhsmtool_posix: 1748867559.612070 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.5' from '/tmp/arc1/sanity-hsm.test_13//0418/0x240000402:0x1a:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.5' lhsmtool_posix: 1748867559.612282 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.4' lhsmtool_posix: 1748867559.633164 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.4' from '/tmp/arc1/sanity-hsm.test_13//0419/0x240000402:0x1b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.4' lhsmtool_posix: 1748867559.633334 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.3' lhsmtool_posix: 1748867559.654245 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.3' from '/tmp/arc1/sanity-hsm.test_13//041e/0x240000402:0x1c:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.3' lhsmtool_posix: 1748867559.654705 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.2' lhsmtool_posix: 1748867559.669261 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.2' from '/tmp/arc1/sanity-hsm.test_13//041f/0x240000402:0x1d:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.2' lhsmtool_posix: 1748867559.669454 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.1' lhsmtool_posix: 1748867559.683060 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.1' from '/tmp/arc1/sanity-hsm.test_13//041c/0x240000402:0x1e:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.1' lhsmtool_posix: 1748867559.690398 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.10' lhsmtool_posix: 1748867559.728155 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.10' from '/tmp/arc1/sanity-hsm.test_13//046d/0x200000402:0x6f:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.10' lhsmtool_posix: 1748867559.728537 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.9' lhsmtool_posix: 1748867559.752986 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.9' from '/tmp/arc1/sanity-hsm.test_13//0472/0x200000402:0x70:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.9' lhsmtool_posix: 1748867559.753322 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.8' lhsmtool_posix: 1748867559.769473 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.8' from '/tmp/arc1/sanity-hsm.test_13//0473/0x200000402:0x71:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.8' lhsmtool_posix: 1748867559.769727 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.7' lhsmtool_posix: 1748867559.784747 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.7' from '/tmp/arc1/sanity-hsm.test_13//0470/0x200000402:0x72:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.7' lhsmtool_posix: 1748867559.784915 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.6' lhsmtool_posix: 1748867559.797870 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.6' from '/tmp/arc1/sanity-hsm.test_13//0471/0x200000402:0x73:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.6' lhsmtool_posix: 1748867559.798053 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.5' lhsmtool_posix: 1748867559.810513 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.5' from '/tmp/arc1/sanity-hsm.test_13//0476/0x200000402:0x74:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.5' lhsmtool_posix: 1748867559.810700 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.4' lhsmtool_posix: 1748867559.829382 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.4' from '/tmp/arc1/sanity-hsm.test_13//0477/0x200000402:0x75:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.4' lhsmtool_posix: 1748867559.829613 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.3' lhsmtool_posix: 1748867559.845179 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.3' from '/tmp/arc1/sanity-hsm.test_13//0474/0x200000402:0x76:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.3' lhsmtool_posix: 1748867559.845325 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.2' lhsmtool_posix: 1748867559.860562 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.2' from '/tmp/arc1/sanity-hsm.test_13//0475/0x200000402:0x77:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.2' lhsmtool_posix: 1748867559.860699 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.1' lhsmtool_posix: 1748867559.879487 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.1' from '/tmp/arc1/sanity-hsm.test_13//047a/0x200000402:0x78:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.1' lhsmtool_posix: 1748867559.881516 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.10' lhsmtool_posix: 1748867559.910391 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.10' from '/tmp/arc1/sanity-hsm.test_13//0422/0x240000402:0x20:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.10' lhsmtool_posix: 1748867559.910544 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.9' lhsmtool_posix: 1748867559.926941 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.9' from '/tmp/arc1/sanity-hsm.test_13//0423/0x240000402:0x21:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.9' lhsmtool_posix: 1748867559.927699 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.8' lhsmtool_posix: 1748867559.944527 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.8' from '/tmp/arc1/sanity-hsm.test_13//0420/0x240000402:0x22:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.8' lhsmtool_posix: 1748867559.944662 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.7' lhsmtool_posix: 1748867559.963933 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.7' from '/tmp/arc1/sanity-hsm.test_13//0421/0x240000402:0x23:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.7' lhsmtool_posix: 1748867559.964353 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.6' lhsmtool_posix: 1748867559.984691 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.6' from '/tmp/arc1/sanity-hsm.test_13//0426/0x240000402:0x24:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.6' lhsmtool_posix: 1748867559.984860 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.5' lhsmtool_posix: 1748867560.004895 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.5' from '/tmp/arc1/sanity-hsm.test_13//0427/0x240000402:0x25:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.5' lhsmtool_posix: 1748867560.005337 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.4' lhsmtool_posix: 1748867560.023744 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.4' from '/tmp/arc1/sanity-hsm.test_13//0424/0x240000402:0x26:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.4' lhsmtool_posix: 1748867560.024027 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.3' lhsmtool_posix: 1748867560.042262 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.3' from '/tmp/arc1/sanity-hsm.test_13//0425/0x240000402:0x27:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.3' lhsmtool_posix: 1748867560.042396 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.2' lhsmtool_posix: 1748867560.060974 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.2' from '/tmp/arc1/sanity-hsm.test_13//042a/0x240000402:0x28:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.2' lhsmtool_posix: 1748867560.061155 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.1' lhsmtool_posix: 1748867560.081887 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.1' from '/tmp/arc1/sanity-hsm.test_13//042b/0x240000402:0x29:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.1' lhsmtool_posix: 1748867560.091103 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.10' lhsmtool_posix: 1748867560.113827 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.10' from '/tmp/arc1/sanity-hsm.test_13//0478/0x200000402:0x7a:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.10' lhsmtool_posix: 1748867560.114049 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.9' lhsmtool_posix: 1748867560.134393 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.9' from '/tmp/arc1/sanity-hsm.test_13//0479/0x200000402:0x7b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.9' lhsmtool_posix: 1748867560.134608 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.8' lhsmtool_posix: 1748867560.155159 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.8' from '/tmp/arc1/sanity-hsm.test_13//047e/0x200000402:0x7c:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.8' lhsmtool_posix: 1748867560.155854 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.7' lhsmtool_posix: 1748867560.175626 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.7' from '/tmp/arc1/sanity-hsm.test_13//047f/0x200000402:0x7d:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.7' lhsmtool_posix: 1748867560.175780 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.6' lhsmtool_posix: 1748867560.194315 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.6' from '/tmp/arc1/sanity-hsm.test_13//047c/0x200000402:0x7e:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.6' lhsmtool_posix: 1748867560.194465 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.5' lhsmtool_posix: 1748867560.209996 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.5' from '/tmp/arc1/sanity-hsm.test_13//047d/0x200000402:0x7f:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.5' lhsmtool_posix: 1748867560.210150 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.4' lhsmtool_posix: 1748867560.234326 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.4' from '/tmp/arc1/sanity-hsm.test_13//0482/0x200000402:0x80:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.4' lhsmtool_posix: 1748867560.234464 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.3' lhsmtool_posix: 1748867560.253765 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.3' from '/tmp/arc1/sanity-hsm.test_13//0483/0x200000402:0x81:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.3' lhsmtool_posix: 1748867560.253926 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.2' lhsmtool_posix: 1748867560.270327 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.2' from '/tmp/arc1/sanity-hsm.test_13//0480/0x200000402:0x82:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.2' lhsmtool_posix: 1748867560.270483 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.1' lhsmtool_posix: 1748867560.293763 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.1' from '/tmp/arc1/sanity-hsm.test_13//0481/0x200000402:0x83:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.1' lhsmtool_posix: 1748867560.295972 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.10' lhsmtool_posix: 1748867560.320524 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.10' from '/tmp/arc1/sanity-hsm.test_13//0429/0x240000402:0x2b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.10' lhsmtool_posix: 1748867560.321219 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.9' lhsmtool_posix: 1748867560.343227 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.9' from '/tmp/arc1/sanity-hsm.test_13//042e/0x240000402:0x2c:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.9' lhsmtool_posix: 1748867560.343379 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.8' lhsmtool_posix: 1748867560.359921 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.8' from '/tmp/arc1/sanity-hsm.test_13//042f/0x240000402:0x2d:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.8' lhsmtool_posix: 1748867560.360135 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.7' lhsmtool_posix: 1748867560.377354 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.7' from '/tmp/arc1/sanity-hsm.test_13//042c/0x240000402:0x2e:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.7' lhsmtool_posix: 1748867560.377583 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.6' lhsmtool_posix: 1748867560.400574 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.6' from '/tmp/arc1/sanity-hsm.test_13//042d/0x240000402:0x2f:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.6' lhsmtool_posix: 1748867560.401038 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.5' lhsmtool_posix: 1748867560.418503 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.5' from '/tmp/arc1/sanity-hsm.test_13//0432/0x240000402:0x30:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.5' lhsmtool_posix: 1748867560.418656 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.4' lhsmtool_posix: 1748867560.435013 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.4' from '/tmp/arc1/sanity-hsm.test_13//0433/0x240000402:0x31:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.4' lhsmtool_posix: 1748867560.435180 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.3' lhsmtool_posix: 1748867560.455892 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.3' from '/tmp/arc1/sanity-hsm.test_13//0430/0x240000402:0x32:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.3' lhsmtool_posix: 1748867560.456056 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.2' lhsmtool_posix: 1748867560.478612 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.2' from '/tmp/arc1/sanity-hsm.test_13//0431/0x240000402:0x33:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.2' lhsmtool_posix: 1748867560.478775 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.1' lhsmtool_posix: 1748867560.497979 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.1' from '/tmp/arc1/sanity-hsm.test_13//0436/0x240000402:0x34:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.1' lhsmtool_posix: 1748867560.508080 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.10' lhsmtool_posix: 1748867560.537253 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.10' from '/tmp/arc1/sanity-hsm.test_13//0487/0x200000402:0x85:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.10' lhsmtool_posix: 1748867560.537658 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.9' lhsmtool_posix: 1748867560.570939 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.9' from '/tmp/arc1/sanity-hsm.test_13//0484/0x200000402:0x86:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.9' lhsmtool_posix: 1748867560.571294 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.8' lhsmtool_posix: 1748867560.597972 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.8' from '/tmp/arc1/sanity-hsm.test_13//0485/0x200000402:0x87:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.8' lhsmtool_posix: 1748867560.598147 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.7' lhsmtool_posix: 1748867560.623022 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.7' from '/tmp/arc1/sanity-hsm.test_13//048a/0x200000402:0x88:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.7' lhsmtool_posix: 1748867560.623263 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.6' lhsmtool_posix: 1748867560.656760 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.6' from '/tmp/arc1/sanity-hsm.test_13//048b/0x200000402:0x89:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.6' lhsmtool_posix: 1748867560.656940 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.5' lhsmtool_posix: 1748867560.681684 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.5' from '/tmp/arc1/sanity-hsm.test_13//0488/0x200000402:0x8a:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.5' lhsmtool_posix: 1748867560.682214 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.4' lhsmtool_posix: 1748867560.698852 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.4' from '/tmp/arc1/sanity-hsm.test_13//0489/0x200000402:0x8b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.4' lhsmtool_posix: 1748867560.699342 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.3' lhsmtool_posix: 1748867560.715591 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.3' from '/tmp/arc1/sanity-hsm.test_13//048e/0x200000402:0x8c:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.3' lhsmtool_posix: 1748867560.716063 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.2' lhsmtool_posix: 1748867560.733454 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.2' from '/tmp/arc1/sanity-hsm.test_13//048f/0x200000402:0x8d:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.2' lhsmtool_posix: 1748867560.736142 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.1' lhsmtool_posix: 1748867560.756052 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.1' from '/tmp/arc1/sanity-hsm.test_13//048c/0x200000402:0x8e:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.1' lhsmtool_posix: 1748867560.759344 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.10' lhsmtool_posix: 1748867560.784026 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.10' from '/tmp/arc1/sanity-hsm.test_13//0434/0x240000402:0x36:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.10' lhsmtool_posix: 1748867560.784294 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.9' lhsmtool_posix: 1748867560.813880 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.9' from '/tmp/arc1/sanity-hsm.test_13//0435/0x240000402:0x37:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.9' lhsmtool_posix: 1748867560.814034 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.8' lhsmtool_posix: 1748867560.830716 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.8' from '/tmp/arc1/sanity-hsm.test_13//043a/0x240000402:0x38:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.8' lhsmtool_posix: 1748867560.830892 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.7' lhsmtool_posix: 1748867560.855097 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.7' from '/tmp/arc1/sanity-hsm.test_13//043b/0x240000402:0x39:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.7' lhsmtool_posix: 1748867560.855282 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.6' lhsmtool_posix: 1748867560.873580 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.6' from '/tmp/arc1/sanity-hsm.test_13//0438/0x240000402:0x3a:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.6' lhsmtool_posix: 1748867560.873748 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.5' lhsmtool_posix: 1748867560.891239 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.5' from '/tmp/arc1/sanity-hsm.test_13//0439/0x240000402:0x3b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.5' lhsmtool_posix: 1748867560.891385 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.4' lhsmtool_posix: 1748867560.911555 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.4' from '/tmp/arc1/sanity-hsm.test_13//043e/0x240000402:0x3c:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.4' lhsmtool_posix: 1748867560.911710 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.3' lhsmtool_posix: 1748867560.929649 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.3' from '/tmp/arc1/sanity-hsm.test_13//043f/0x240000402:0x3d:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.3' lhsmtool_posix: 1748867560.929796 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.2' lhsmtool_posix: 1748867560.944747 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.2' from '/tmp/arc1/sanity-hsm.test_13//043c/0x240000402:0x3e:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.2' lhsmtool_posix: 1748867560.944910 lhsmtool_posix[6448]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.1' lhsmtool_posix: 1748867560.958286 lhsmtool_posix[6448]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.1' from '/tmp/arc1/sanity-hsm.test_13//043d/0x240000402:0x3f:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.1' lhsmtool_posix: 1748867560.958388 lhsmtool_posix[6448]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_13/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' PASS 13 (10s) == sanity-hsm test 14: Rebind archived file to a new fid ========================================================== 08:32:46 (1748867566) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_14/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.042461 s, 24.7 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00160532 s, 0.0 kB/s rebind 0x200000403:0x73:0x0 to 0x200000403:0x74:0x0 lhsmtool_posix: 1748867568.468805 lhsmtool_posix[7481]: action=2 src=0x200000403:0x73:0x0 dst=0x200000403:0x74:0x0 mount_point=/mnt/lustre2 lhsmtool_posix: 1748867568.470612 lhsmtool_posix[7481]: rebind [0x200000403:0x73:0x0] to [0x200000403:0x74:0x0] lhsmtool_posix: 1748867568.470754 lhsmtool_posix[7481]: process finished, errs: 0 major, 0 minor, rc=0 (Success) PASS 14 (4s) == sanity-hsm test 15: Rebind a list of files ============ 08:32:50 (1748867570) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_15/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0473719 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0522254 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0382158 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0434323 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0401933 s, 26.1 MB/s pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00293868 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00325314 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00239862 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0016812 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00192636 s, 0.0 kB/s rebind list of files lhsmtool_posix: 1748867572.578455 lhsmtool_posix[8319]: action=2 src=/tmp/tmp.7959 dst=(null) mount_point=/mnt/lustre2 lhsmtool_posix: 1748867572.579632 lhsmtool_posix[8319]: rebind [0x200000403:0x76:0x0] to [0x200000403:0x7b:0x0] lhsmtool_posix: 1748867572.579745 lhsmtool_posix[8319]: rebind [0x200000403:0x77:0x0] to [0x200000403:0x7c:0x0] lhsmtool_posix: 1748867572.579807 lhsmtool_posix[8319]: rebind [0x200000403:0x78:0x0] to [0x200000403:0x7d:0x0] lhsmtool_posix: 1748867572.579846 lhsmtool_posix[8319]: rebind [0x200000403:0x79:0x0] to [0x200000403:0x7e:0x0] lhsmtool_posix: 1748867572.579880 lhsmtool_posix[8319]: rebind [0x200000403:0x7a:0x0] to [0x200000403:0x7f:0x0] lhsmtool_posix: 1748867572.579929 lhsmtool_posix[8319]: 5 lines read from '/tmp/tmp.7959', 5 rebind successful lhsmtool_posix: 1748867572.579961 lhsmtool_posix[8319]: process finished, errs: 0 major, 0 minor, rc=0 (Success) PASS 15 (9s) == sanity-hsm test 16: Test CT bandwith control option === 08:32:59 (1748867579) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_16/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' 20+0 records in 20+0 records out 20971520 bytes (21 MB) copied, 0.0328209 s, 639 MB/s Waiting 200s for 'SUCCEED' Waiting 180s for 'SUCCEED' Updated after 21s: want 'SUCCEED' got 'SUCCEED' PASS 16 (24s) == sanity-hsm test 20: Release is not permitted ========== 08:33:23 (1748867603) 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00190695 s, 0.0 kB/s 0x200000403:0x87:0x0 Cannot send HSM request (use of /mnt/lustre/d20.sanity-hsm/f20.sanity-hsm): Operation not permitted Cannot send HSM request (use of /mnt/lustre/d20.sanity-hsm/f20.sanity-hsm): Operation not permitted Cannot send HSM request (use of /mnt/lustre/d20.sanity-hsm/f20.sanity-hsm): Operation not permitted Cannot send HSM request (use of /mnt/lustre/d20.sanity-hsm/f20.sanity-hsm): Operation not permitted PASS 20 (3s) == sanity-hsm test 21: Simple release tests ============== 08:33:26 (1748867606) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_21/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0928741 s, 11.3 MB/s Waiting 200s for 'SUCCEED' PASS 21 (5s) == sanity-hsm test 22: Could not swap a release file ===== 08:33:31 (1748867611) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_22/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0439049 s, 23.9 MB/s Waiting 200s for 'SUCCEED' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0610956 s, 17.2 MB/s 0x200000403:0x8b:0x0 lfs swap_layouts: error: cannot swap layout between '/mnt/lustre/d22.sanity-hsm/test_swap' and '/mnt/lustre/d22.sanity-hsm/test_release': Operation not supported (95) PASS 22 (4s) == sanity-hsm test 23: Release does not change a/mtime (utime) ========================================================== 08:33:35 (1748867615) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_23/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0701835 s, 14.9 MB/s PASS 23 (4s) == sanity-hsm test 24a: Archive, release, and restore does not change a/mtime (i/o) ========================================================== 08:33:39 (1748867619) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0470182 s, 22.3 MB/s Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' Killing existing copytools on oleg226-client.virtnet copytools stopped in 0s 192.168.202.126@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg226-client.virtnet /mnt/lustre (opts:) Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre PASS 24a (8s) == sanity-hsm test 24b: root can archive, release, and restore user files ========================================================== 08:33:47 (1748867627) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0659384 s, 15.9 MB/s Waiting 200s for 'SUCCEED' running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d24b.sanity-hsm/f24b.sanity-hsm] /mnt/lustre/d24b.sanity-hsm/f24b.sanity-hsm: (0x00000009) exists archived, archive_id:1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [md5sum] [/mnt/lustre/d24b.sanity-hsm/f24b.sanity-hsm] PASS 24b (7s) == sanity-hsm test 24c: check that user,group,other request masks work ========================================================== 08:33:54 (1748867634) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0647817 s, 16.2 MB/s 0x200000403:0x94:0x0 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_archive] [/mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm] Cannot send HSM request (use of /mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm): Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_archive] [/mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.072768 s, 14.4 MB/s 0x200000403:0x95:0x0 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_archive] [/mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm] Cannot send HSM request (use of /mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm): Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_archive] [/mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0684687 s, 15.3 MB/s 0x200000403:0x96:0x0 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_archive] [/mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm] Cannot send HSM request (use of /mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm): Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_archive] [/mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm] PASS 24c (9s) == sanity-hsm test 24d: check that read-only mounts are respected ========================================================== 08:34:03 (1748867643) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0472923 s, 22.2 MB/s 0x200000403:0x97:0x0 /mnt/lustre/d24d.sanity-hsm/f24d.sanity-hsm lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 81 0x51 0x2c0000401 Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre3 Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre3"' Cannot send HSM request (use of /mnt/lustre2/d24d.sanity-hsm/f24d.sanity-hsm): Read-only file system Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.510425 s, 2.1 MB/s Cannot send HSM request (use of /mnt/lustre2/d24d.sanity-hsm/f24d.sanity-hsm): Read-only file system Stopping client oleg226-client.virtnet /mnt/lustre3 (opts:) PASS 24d (8s) == sanity-hsm test 24e: tar succeeds on HSM released files ========================================================== 08:34:11 (1748867651) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24e/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0632054 s, 16.6 MB/s Waiting 200s for 'SUCCEED' /mnt/lustre/d24e.sanity-hsm/f24e.sanity-hsm: (0x0000000d) released exists archived, archive_id:1 tar: Removing leading `/' from member names PASS 24e (5s) == sanity-hsm test 24f: root can archive, release, and restore tar files ========================================================== 08:34:16 (1748867656) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24f/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 3d2fd8331483d30d32d70431b70233ef /mnt/lustre/d24f.sanity-hsm/f24f.sanity-hsm Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' f24f.sanity-hsm f24f.sanity-hsm Sum0 = 3d2fd8331483d30d32d70431b70233ef /mnt/lustre/d24f.sanity-hsm/f24f.sanity-hsm, sum1 = 3d2fd8331483d30d32d70431b70233ef /mnt/lustre/d24f.sanity-hsm/f24f.sanity-hsm PASS 24f (5s) == sanity-hsm test 24g: write by non-owner still sets dirty ========================================================== 08:34:21 (1748867661) RUNAS = 'runas -u 500 -g 500' Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24g/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Please listen carefully as our options have changed. Waiting 200s for 'SUCCEED' running as uid/gid/euid/egid 500/500/500/500, groups: 500 [tee] [/mnt/lustre/d24g.sanity-hsm/f24g.sanity-hsm] To be electrocuted by your telephone, press #. PASS 24g (4s) == sanity-hsm test 25a: Restore lost file (HS_LOST flag) from import ========================================================== 08:34:25 (1748867665) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_25a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1748867666.845303 lhsmtool_posix[18197]: action=1 src=d25a.sanity-hsm/f25a.sanity-hsm dst=/mnt/lustre/d25a.sanity-hsm/f25a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1748867666.853798 lhsmtool_posix[18197]: importing '/mnt/lustre/d25a.sanity-hsm/f25a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_25a//d25a.sanity-hsm/f25a.sanity-hsm' lhsmtool_posix: 1748867666.899563 lhsmtool_posix[18197]: imported '/mnt/lustre/d25a.sanity-hsm/f25a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_25a//0406/0x240000404:0x2:0x0'=='/tmp/arc1/sanity-hsm.test_25a//d25a.sanity-hsm/f25a.sanity-hsm' lhsmtool_posix: 1748867666.900603 lhsmtool_posix[18197]: process finished, errs: 0 major, 0 minor, rc=0 (Success) md5sum: /mnt/lustre/d25a.sanity-hsm/f25a.sanity-hsm: No data available PASS 25a (4s) == sanity-hsm test 25b: Restore lost file (HS_LOST flag) after release ========================================================== 08:34:29 (1748867669) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_25b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' md5sum: /mnt/lustre/d25b.sanity-hsm/f25b.sanity-hsm: No data available PASS 25b (5s) == sanity-hsm test 26A: Remove the archive of a valid file ========================================================== 08:34:34 (1748867674) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26A/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.002142 s, 0.0 kB/s Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' PASS 26A (6s) == sanity-hsm test 26a: Remove Archive On Last Unlink (RAoLU) policy ========================================================== 08:34:40 (1748867680) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Waiting 200s for 'SUCCEED' Updated after 3s: want 'SUCCEED' got 'SUCCEED' PASS 26a (17s) == sanity-hsm test 26b: RAoLU policy when CDT off ======== 08:34:57 (1748867697) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled Killing existing copytools on oleg226-client.virtnet copytools stopped in 0s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' PASS 26b (12s) == sanity-hsm test 26c: RAoLU effective when file closed ========================================================== 08:35:09 (1748867709) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' multiop /mnt/lustre/d26c.sanity-hsm/f26c.sanity-hsm vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7959 Waiting 200s for 'SUCCEED' Updated after 9s: want 'SUCCEED' got 'SUCCEED' Waiting 200s for 'SUCCEED' Updated after 8s: want 'SUCCEED' got 'SUCCEED' PASS 26c (30s) == sanity-hsm test 26d: RAoLU when Client eviction ======= 08:35:39 (1748867739) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0630263 s, 16.6 MB/s Waiting 200s for 'SUCCEED' multiop /mnt/lustre/d26d.sanity-hsm/f26d.sanity-hsm vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7959 Waiting 200s for 'SUCCEED' Updated after 9s: want 'SUCCEED' got 'SUCCEED' PASS 26d (21s) == sanity-hsm test 26e: RAoLU with a non-started coordinator ========================================================== 08:36:00 (1748867760) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26e/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0574855 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0537359 s, 19.5 MB/s Waiting 200s for 'SUCCEED' Killing existing copytools on oleg226-client.virtnet copytools stopped in 0s Failing mds1 on oleg226-server Stopping /mnt/lustre-mds1 (opts:) on oleg226-server 08:36:26 (1748867786) shut down facet: mds1 facet_host: oleg226-server facet_failover_host: oleg226-server Failover mds1 to oleg226-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0000 08:36:40 (1748867800) targets are mounted 08:36:40 (1748867800) facet_failover done oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Check cookie from RAoLU request (last: 1748867493, remove: 1748867494) mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26e/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' PASS 26e (75s) == sanity-hsm test 27a: Remove the archive of an imported file (Operation not permitted) ========================================================== 08:37:15 (1748867835) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_27a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 39+0 records in 39+0 records out 40894464 bytes (41 MB) copied, 0.461717 s, 88.6 MB/s lhsmtool_posix: 1748867836.457192 lhsmtool_posix[26025]: action=1 src=d27a.sanity-hsm/f27a.sanity-hsm dst=/mnt/lustre/d27a.sanity-hsm/f27a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1748867836.460626 lhsmtool_posix[26025]: importing '/mnt/lustre/d27a.sanity-hsm/f27a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_27a//d27a.sanity-hsm/f27a.sanity-hsm' lhsmtool_posix: 1748867836.497678 lhsmtool_posix[26025]: imported '/mnt/lustre/d27a.sanity-hsm/f27a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_27a//0400/0x240000404:0x4:0x0'=='/tmp/arc1/sanity-hsm.test_27a//d27a.sanity-hsm/f27a.sanity-hsm' lhsmtool_posix: 1748867836.497747 lhsmtool_posix[26025]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Cannot send HSM request (use of /mnt/lustre/d27a.sanity-hsm/f27a.sanity-hsm): Operation not permitted PASS 27a (3s) == sanity-hsm test 27b: Remove the archive of a relased file (Operation not permitted) ========================================================== 08:37:18 (1748867838) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_27b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0028868 s, 0.0 kB/s Waiting 200s for 'SUCCEED' Cannot send HSM request (use of /mnt/lustre/d27b.sanity-hsm/f27b.sanity-hsm): Operation not permitted PASS 27b (4s) == sanity-hsm test 28: Concurrent archive/file remove ==== 08:37:22 (1748867842) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_28/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00191697 s, 0.0 kB/s Waiting 200s for 'SUCCEED' mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 28 (7s) == sanity-hsm test 29a: Tests --mntpath and --archive options ========================================================== 08:37:29 (1748867849) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_29a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 7 "/mnt/lustre"' Cannot send HSM request (use of /mnt/lustre): Invalid argument hsm: '/qwerty/uyt' is not a valid FID PASS 29a (2s) == sanity-hsm test 29b: Archive/delete/remove by FID from the archive. ========================================================== 08:37:31 (1748867851) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_29b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0386814 s, 27.1 MB/s Waiting 200s for 'SUCCEED' PASS 29b (4s) == sanity-hsm test 29c: Archive/delete/remove by FID, using a file list. ========================================================== 08:37:35 (1748867855) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_29c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0435289 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0435166 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0435565 s, 24.1 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Waiting 200s for 'SUCCEED' PASS 29c (7s) == sanity-hsm test 29d: hsm_remove by FID with archive_id 0 for unlinked file cause ========================================================== 08:37:43 (1748867863) SKIP: sanity-hsm test_29d Need 3 or more clients, have 1 SKIP 29d (0s) == sanity-hsm test 30a: Restore at exec (import case) ==== 08:37:43 (1748867863) SKIP: sanity-hsm test_30a Need 2 or more clients, have 1 SKIP 30a (1s) == sanity-hsm test 30b: Restore at exec (release case) === 08:37:44 (1748867864) SKIP: sanity-hsm test_30b Need 2 or more clients, have 1 SKIP 30b (1s) == sanity-hsm test 30c: Update during exec of released file must fail ========================================================== 08:37:45 (1748867865) SKIP: sanity-hsm test_30c Need 2 or more clients, have 1 SKIP 30c (1s) == sanity-hsm test 31a: Import a large file and check size during restore ========================================================== 08:37:46 (1748867866) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_31a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 39+0 records in 39+0 records out 40894464 bytes (41 MB) copied, 0.419833 s, 97.4 MB/s lhsmtool_posix: 1748867867.444760 lhsmtool_posix[31669]: action=1 src=d31a.sanity-hsm/f31a.sanity-hsm dst=/mnt/lustre/d31a.sanity-hsm/f31a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1748867867.446959 lhsmtool_posix[31669]: importing '/mnt/lustre/d31a.sanity-hsm/f31a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_31a//d31a.sanity-hsm/f31a.sanity-hsm' lhsmtool_posix: 1748867867.456132 lhsmtool_posix[31669]: imported '/mnt/lustre/d31a.sanity-hsm/f31a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_31a//040c/0x200000407:0xb:0x0'=='/tmp/arc1/sanity-hsm.test_31a//d31a.sanity-hsm/f31a.sanity-hsm' lhsmtool_posix: 1748867867.456220 lhsmtool_posix[31669]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_31a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' size seen is right: 40894464 == 40894464 done PASS 31a (13s) == sanity-hsm test 31b: Restore a large unaligned file and check size during restore ========================================================== 08:37:59 (1748867879) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_31b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 39+0 records in 39+0 records out 39000000 bytes (39 MB) copied, 0.306162 s, 127 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' size seen is right: 39000000 == 39000000 done PASS 31b (14s) == sanity-hsm test 31c: Restore a large aligned file and check size during restore ========================================================== 08:38:13 (1748867893) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_31c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 39+0 records in 39+0 records out 40894464 bytes (41 MB) copied, 0.280174 s, 146 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' size seen is right: 40894464 == 40894464 done PASS 31c (13s) == sanity-hsm test 33: Kill a restore waiting process ==== 08:38:26 (1748867906) 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00136916 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_33/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Copytool is suspended on oleg226-client.virtnet Copytool is continued on oleg226-client.virtnet /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh: line 2829: 1469 Terminated md5sum $f > /dev/null PASS 33 (4s) == sanity-hsm test 34: Remove file during restore ======== 08:38:30 (1748867910) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_34/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00141247 s, 0.0 kB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Copytool is suspended on oleg226-client.virtnet Waiting 200s for 'STARTED' Copytool is continued on oleg226-client.virtnet /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh: line 2861: kill: (2214) - No such process PASS 34 (5s) == sanity-hsm test 35: Overwrite file during restore ===== 08:38:35 (1748867915) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_35/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00147588 s, 0.0 kB/s Waiting 200s for 'SUCCEED' Copytool is suspended on oleg226-client.virtnet Waiting 200s for 'STARTED' Updated after 2s: want 'STARTED' got 'STARTED' Copytool is continued on oleg226-client.virtnet /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh: line 2898: kill: (2966) - No such process PASS 35 (5s) == sanity-hsm test 36: Move file during restore ========== 08:38:40 (1748867920) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_36/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00196135 s, 0.0 kB/s Waiting 200s for 'SUCCEED' Copytool is suspended on oleg226-client.virtnet Waiting 200s for 'STARTED' Copytool is continued on oleg226-client.virtnet /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh: line 2934: kill: (3715) - No such process PASS 36 (6s) == sanity-hsm test 37: re-archive a dirty file =========== 08:38:46 (1748867926) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_37/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0453908 s, 23.1 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0219708 s, 47.7 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' PASS 37 (66s) == sanity-hsm test 40: Parallel archive requests ========= 08:39:53 (1748867993) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_40/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge Wait for all requests being enqueued...100 archive requests submitted 100 archive requests submitted 100 archive requests submitted 100 archive requests submitted OK pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 PASS 40 (88s) == sanity-hsm test 50: Archive with large number of pending HSM actions ========================================================== 08:41:20 (1748868080) Filesystem Inodes IUsed IFree IUse% Mounted on 192.168.202.126@tcp:/lustre 524226 1219 523007 1% /mnt/lustre - create 6860 (time 1748868092.34 total 10.00 last 685.97) total: 10000 create in 14.51 seconds: 689.26 ops/second 10000 - unlinked 0 (time 1748868116 ; total 0 ; last 0) total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second Start Phase files_num: 10000 time: 18 - create 7198 (time 1748868143.08 total 10.00 last 719.76) - create 10000 (time 1748868146.70 total 13.63 last 772.86) - create 16685 (time 1748868156.70 total 23.63 last 668.44) total: 20000 create in 27.98 seconds: 714.87 ops/second - unlinked 0 (time 1748868199 ; total 0 ; last 0) - unlinked 10000 (time 1748868218 ; total 19 ; last 19) total: 20000 unlinks in 33 seconds: 606.060608 unlinks/second Middle Phase files_num: 20000 time: 38 - create 7789 (time 1748868242.92 total 10.00 last 778.89) total: 10000 create in 12.88 seconds: 776.47 ops/second 40000 - unlinked 0 (time 1748868268 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second End Phase files_num: 10000 time: 20 40000 mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge PASS 50 (203s) == sanity-hsm test 52: Opened for write file on an evicted client should be set dirty ========================================================== 08:44:43 (1748868283) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_52/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0317952 s, 33.0 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' multiop /mnt/lustre/d52.sanity-hsm/f52.sanity-hsm vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7959 pdsh@oleg226-client: oleg226-client: ssh exited with exit code 5 PASS 52 (7s) == sanity-hsm test 53: Opened for read file on an evicted client should not be set dirty ========================================================== 08:44:50 (1748868290) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_53/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0391922 s, 26.8 MB/s multiop /mnt/lustre/d53.sanity-hsm/f53.sanity-hsm vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7959 pdsh@oleg226-client: oleg226-client: ssh exited with exit code 5 PASS 53 (6s) == sanity-hsm test 54: Write during an archive cancels it ========================================================== 08:44:56 (1748868296) 39+0 records in 39+0 records out 39000000 bytes (39 MB) copied, 0.29952 s, 130 MB/s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_54/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' Waiting 200s for 'STARTED' Updated after 2s: want 'STARTED' got 'STARTED' mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0001.hsm.policy=+NRA Waiting 200s for 'FAILED' Waiting 190s for 'FAILED' Waiting 180s for 'FAILED' Waiting 170s for 'FAILED' Updated after 35s: want 'FAILED' got 'FAILED' mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA PASS 54 (42s) == sanity-hsm test 55: Truncate during an archive cancels it ========================================================== 08:45:38 (1748868338) 39+0 records in 39+0 records out 39000000 bytes (39 MB) copied, 0.286368 s, 136 MB/s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_55/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0001.hsm.policy=+NRA mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA PASS 55 (4s) == sanity-hsm test 56: Setattr during an archive is ok === 08:45:43 (1748868343) 39+0 records in 39+0 records out 39000000 bytes (39 MB) copied, 0.32411 s, 120 MB/s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_56/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' Waiting 200s for 'STARTED' Updated after 2s: want 'STARTED' got 'STARTED' Waiting 200s for 'SUCCEED' Waiting 190s for 'SUCCEED' Waiting 180s for 'SUCCEED' Waiting 170s for 'SUCCEED' Updated after 36s: want 'SUCCEED' got 'SUCCEED' PASS 56 (41s) == sanity-hsm test 57: Archive a file with dirty cache on another node ========================================================== 08:46:23 (1748868383) SKIP: sanity-hsm test_57 Need 2 or more clients, have 1 SKIP 57 (1s) == sanity-hsm test 58: Truncate a released file will trigger restore ========================================================== 08:46:24 (1748868384) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_58/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' truncate up from 1716 to 3432 Waiting 200s for 'SUCCEED' /mnt/lustre/d58.sanity-hsm/f58.sanity-hsm: (0x0000000b) exists dirty archived, archive_id:2 truncate down from 1716 to 858 Waiting 200s for 'SUCCEED' /mnt/lustre/d58.sanity-hsm/f58.sanity-hsm: (0x0000000b) exists dirty archived, archive_id:2 truncate to 0 Waiting 200s for 'SUCCEED' /mnt/lustre/d58.sanity-hsm/f58.sanity-hsm: (0x0000000b) exists dirty archived, archive_id:2 PASS 58 (8s) == sanity-hsm test 59: Release stripeless file with non-zero size ========================================================== 08:46:32 (1748868392) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_59/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' PASS 59 (4s) == sanity-hsm test 60: Changing progress update interval from default ========================================================== 08:46:36 (1748868396) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0848099 s, 124 MB/s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_60/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 --update-interval 5 "/mnt/lustre2"' Waiting 100s for 'start copy' Expecting a progress update within 20 seconds... 5242880 bytes copied in 4 seconds. Wait for on going archive hsm action to complete Waiting 10s for 'copied' Updated after 4s: want 'copied' got 'copied' PASS 60 (12s) == sanity-hsm test 61: Waiting archive of a removed file should fail ========================================================== 08:46:49 (1748868409) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_61/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 61 (5s) == sanity-hsm test 70: Copytool logs JSON register/unregister events to FIFO ========================================================== 08:46:53 (1748868413) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_70/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --event-fifo /tmp/sanity-hsm.test_70.y2vx/fifo "/mnt/lustre2"' Killing existing copytools on oleg226-client.virtnet copytools stopped in 0s Register/Unregister events look OK. PASS 70 (3s) == sanity-hsm test 71: Copytool logs JSON archive events to FIFO ========================================================== 08:46:56 (1748868416) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_71/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --update-interval 5 --event-fifo /tmp/sanity-hsm.test_71.qOAJ/fifo "/mnt/lustre2"' mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0318405 s, 32.9 MB/s Waiting 200s for 'SUCCEED' Archive events look OK. mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA PASS 71 (6s) == sanity-hsm test 72: Copytool logs JSON restore events to FIFO ========================================================== 08:47:03 (1748868423) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_72/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --update-interval 5 --event-fifo /tmp/sanity-hsm.test_72.xAbm/fifo "/mnt/lustre2"' 16+0 records in 16+0 records out 16000000 bytes (16 MB) copied, 0.14789 s, 108 MB/s lhsmtool_posix: 1748868424.809554 lhsmtool_posix[22729]: action=1 src=d72.sanity-hsm/f72.sanity-hsm dst=/mnt/lustre/d72.sanity-hsm/f72.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1748868424.811712 lhsmtool_posix[22729]: importing '/mnt/lustre/d72.sanity-hsm/f72.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_72//d72.sanity-hsm/f72.sanity-hsm' lhsmtool_posix: 1748868424.820012 lhsmtool_posix[22729]: imported '/mnt/lustre/d72.sanity-hsm/f72.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_72//0bc5/0x200000bd2:0x17:0x0'=='/tmp/arc1/sanity-hsm.test_72//d72.sanity-hsm/f72.sanity-hsm' lhsmtool_posix: 1748868424.820062 lhsmtool_posix[22729]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Verifying released state: Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Restore events look OK. PASS 72 (6s) == sanity-hsm test 90: Archive/restore a file list ======= 08:47:08 (1748868428) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_90/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge Waiting 200s for '' pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Waiting 200s for '' pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 PASS 90 (70s) == sanity-hsm test 100: Set coordinator /proc tunables === 08:48:18 (1748868498) Testing mdt.lustre-MDT0000.hsm.loop_period Set loop_period to 0 must failed oleg226-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/hsm/loop_period=0: Invalid argument oleg226-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/loop_period'='0': Invalid argument pdsh@oleg226-client: oleg226-server: ssh exited with exit code 22 oleg226-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0001/hsm/loop_period=0: Invalid argument oleg226-server: error: set_param: setting 'mdt/lustre-MDT0001/hsm/loop_period'='0': Invalid argument pdsh@oleg226-client: oleg226-server: ssh exited with exit code 22 Testing mdt.lustre-MDT0000.hsm.grace_delay Set grace_delay to 0 must failed oleg226-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/hsm/grace_delay=0: Invalid argument oleg226-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/grace_delay'='0': Invalid argument pdsh@oleg226-client: oleg226-server: ssh exited with exit code 22 oleg226-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0001/hsm/grace_delay=0: Invalid argument oleg226-server: error: set_param: setting 'mdt/lustre-MDT0001/hsm/grace_delay'='0': Invalid argument pdsh@oleg226-client: oleg226-server: ssh exited with exit code 22 Testing mdt.lustre-MDT0000.hsm.active_request_timeout Set active_request_timeout to 0 must failed oleg226-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/hsm/active_request_timeout=0: Invalid argument oleg226-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/active_request_timeout'='0': Invalid argument pdsh@oleg226-client: oleg226-server: ssh exited with exit code 22 oleg226-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0001/hsm/active_request_timeout=0: Invalid argument oleg226-server: error: set_param: setting 'mdt/lustre-MDT0001/hsm/active_request_timeout'='0': Invalid argument pdsh@oleg226-client: oleg226-server: ssh exited with exit code 22 Testing mdt.lustre-MDT0000.hsm.max_requests Set max_requests to 0 must failed oleg226-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/hsm/max_requests=0: Invalid argument oleg226-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/max_requests'='0': Invalid argument pdsh@oleg226-client: oleg226-server: ssh exited with exit code 22 oleg226-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0001/hsm/max_requests=0: Invalid argument oleg226-server: error: set_param: setting 'mdt/lustre-MDT0001/hsm/max_requests'='0': Invalid argument pdsh@oleg226-client: oleg226-server: ssh exited with exit code 22 Testing mdt.lustre-MDT0000.hsm.default_archive_id Set default_archive_id to 0 must failed oleg226-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/hsm/default_archive_id=0: Invalid argument oleg226-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/default_archive_id'='0': Invalid argument pdsh@oleg226-client: oleg226-server: ssh exited with exit code 22 oleg226-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0001/hsm/default_archive_id=0: Invalid argument oleg226-server: error: set_param: setting 'mdt/lustre-MDT0001/hsm/default_archive_id'='0': Invalid argument pdsh@oleg226-client: oleg226-server: ssh exited with exit code 22 PASS 100 (13s) == sanity-hsm test 102: Verify coordinator control ======= 08:48:31 (1748868511) mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 102 (6s) == sanity-hsm test 103: Purge all requests =============== 08:48:37 (1748868517) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_103/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge Current requests pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 PASS 103 (4s) == sanity-hsm test 103a: Purge pending restore requests == 08:48:41 (1748868521) mdt.lustre-MDT0000.hsm.policy=-NBR mdt.lustre-MDT0001.hsm.policy=-NBR Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_103a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' (0x200000403:0x148:0x0|0x200000403:0x149:0x0|0x200000403:0x14a:0x0|0x200000403:0x14b:0x0|0x200000403:0x14c:0x0|0x200000403:0x14d:0x0|0x200000403:0x14e:0x0|0x200000403:0x14f:0x0).*action=ARCHIVE.*status=SUCCEED Killing existing copytools on oleg226-client.virtnet copytools stopped in 0s cat: /mnt/lustre/.lustre/fid/0x200000403:0x14c:0x0: No data available cat: /mnt/lustre/.lustre/fid/0x200000403:0x148:0x0: No data available cat: /mnt/lustre/.lustre/fid/0x200000403:0x149:0x0cat: /mnt/lustre/.lustre/fid/0x200000403:0x14a:0x0: No data available : No data available cat: /mnt/lustre/.lustre/fid/0x200000403:0x14b:0x0: No data available cat: /mnt/lustre/.lustre/fid/0x200000403:0x14d:0x0: No data available cat: /mnt/lustre/.lustre/fid/0x200000403:0x14e:0x0: No data available cat: /mnt/lustre/.lustre/fid/0x200000403:0x14f:0x0: No data available mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge PASS 103a (8s) == sanity-hsm test 104: Copy tool data field ============= 08:48:49 (1748868529) 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00140592 s, 0.0 kB/s mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge PASS 104 (5s) == sanity-hsm test 105: Restart of coordinator =========== 08:48:54 (1748868534) mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge PASS 105 (13s) == sanity-hsm test 106: Copytool register/unregister ===== 08:49:07 (1748868547) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_106/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' oleg226-client.virtnet: executing pgrep --pidfile=/var/run/lhsmtool_posix.pid --list-full hsmtool oleg226-client.virtnet: executing get_client_uuid /mnt/lustre2 found agent uuid=c8af5ef9-f4c9-4606-93aa-65a65d478e84 archive_id=ANY requests=[current:0 ok:0 errors:0] on mds1 found agent uuid=c8af5ef9-f4c9-4606-93aa-65a65d478e84 archive_id=ANY requests=[current:0 ok:0 errors:0] on mds2 oleg226-client.virtnet: 29836 Killing existing copytools on oleg226-client.virtnet copytools stopped in 0s pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 uuid not found in agent list on mds1 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 uuid not found in agent list on mds2 Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_106/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' oleg226-client.virtnet: executing pgrep --pidfile=/var/run/lhsmtool_posix.pid --list-full hsmtool oleg226-client.virtnet: executing get_client_uuid /mnt/lustre2 found agent uuid=c8af5ef9-f4c9-4606-93aa-65a65d478e84 archive_id=ANY requests=[current:0 ok:0 errors:0] on mds1 found agent uuid=c8af5ef9-f4c9-4606-93aa-65a65d478e84 archive_id=ANY requests=[current:0 ok:0 errors:0] on mds2 PASS 106 (8s) == sanity-hsm test 107: Copytool re-register after MDS restart ========================================================== 08:49:15 (1748868555) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_107/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Failing mds1 on oleg226-server Stopping /mnt/lustre-mds1 (opts:) on oleg226-server 08:49:19 (1748868559) shut down facet: mds1 facet_host: oleg226-server facet_failover_host: oleg226-server Failover mds1 to oleg226-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0000 08:49:33 (1748868573) targets are mounted 08:49:33 (1748868573) facet_failover done oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Waiting 200s for 'SUCCEED' PASS 107 (27s) == sanity-hsm test 109: Policy display/change ============ 08:49:42 (1748868582) mdt.lustre-MDT0000.hsm.policy=+NBR mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0000.hsm.policy=-NBR mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0000.hsm.policy=NRA NBR Next set_param must failed oleg226-server: error: set_param: setting /sys/kernel/debug/lustre/mdt/lustre-MDT0000/hsm/policy=wrong: Invalid argument oleg226-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/policy'='wrong': Invalid argument pdsh@oleg226-client: oleg226-server: ssh exited with exit code 22 Back to default policy mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0001.hsm.policy=+NRA mdt.lustre-MDT0000.hsm.policy=-NBR mdt.lustre-MDT0001.hsm.policy=-NBR PASS 109 (6s) == sanity-hsm test 110a: Non blocking restore policy (import case) ========================================================== 08:49:48 (1748868588) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_110a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1748868589.301513 lhsmtool_posix[1547]: action=1 src=d110a.sanity-hsm/f110a.sanity-hsm dst=/mnt/lustre/d110a.sanity-hsm/f110a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1748868589.304380 lhsmtool_posix[1547]: importing '/mnt/lustre/d110a.sanity-hsm/f110a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_110a//d110a.sanity-hsm/f110a.sanity-hsm' lhsmtool_posix: 1748868589.320736 lhsmtool_posix[1547]: imported '/mnt/lustre/d110a.sanity-hsm/f110a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_110a//0bb6/0x200000bd2:0x64:0x0'=='/tmp/arc1/sanity-hsm.test_110a//d110a.sanity-hsm/f110a.sanity-hsm' lhsmtool_posix: 1748868589.320822 lhsmtool_posix[1547]: process finished, errs: 0 major, 0 minor, rc=0 (Success) mdt.lustre-MDT0000.hsm.policy=+NBR mdt.lustre-MDT0001.hsm.policy=+NBR md5sum: /mnt/lustre/d110a.sanity-hsm/f110a.sanity-hsm: No data available Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' mdt.lustre-MDT0000.hsm.policy=-NBR mdt.lustre-MDT0001.hsm.policy=-NBR PASS 110a (5s) == sanity-hsm test 110b: Non blocking restore policy (release case) ========================================================== 08:49:53 (1748868593) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_110b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' mdt.lustre-MDT0000.hsm.policy=+NBR mdt.lustre-MDT0001.hsm.policy=+NBR md5sum: /mnt/lustre/d110b.sanity-hsm/f110b.sanity-hsm: No data available Waiting 200s for 'SUCCEED' mdt.lustre-MDT0000.hsm.policy=-NBR mdt.lustre-MDT0001.hsm.policy=-NBR PASS 110b (6s) == sanity-hsm test 111a: No retry policy (import case), restore will error ========================================================== 08:49:59 (1748868599) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_111a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1748868599.855699 lhsmtool_posix[3096]: action=1 src=d111a.sanity-hsm/f111a.sanity-hsm dst=/mnt/lustre/d111a.sanity-hsm/f111a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1748868599.858418 lhsmtool_posix[3096]: importing '/mnt/lustre/d111a.sanity-hsm/f111a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_111a//d111a.sanity-hsm/f111a.sanity-hsm' lhsmtool_posix: 1748868599.869263 lhsmtool_posix[3096]: imported '/mnt/lustre/d111a.sanity-hsm/f111a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_111a//0bba/0x200000bd2:0x68:0x0'=='/tmp/arc1/sanity-hsm.test_111a//d111a.sanity-hsm/f111a.sanity-hsm' lhsmtool_posix: 1748868599.869322 lhsmtool_posix[3096]: process finished, errs: 0 major, 0 minor, rc=0 (Success) mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0001.hsm.policy=+NRA Remove from backend: 0x200000bd2:0x68:0x0 = /tmp/arc1/sanity-hsm.test_111a/0bba/0x200000bd2:0x68:0x0 Waiting 200s for 'FAILED' mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA PASS 111a (4s) == sanity-hsm test 111b: No retry policy (release case), restore will error ========================================================== 08:50:03 (1748868603) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_111b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0001.hsm.policy=+NRA Waiting 200s for 'SUCCEED' Remove from backend: 0x200000403:0x158:0x0 = /tmp/arc1/sanity-hsm.test_111b/055b/0x200000403:0x158:0x0 Waiting 200s for 'FAILED' mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA PASS 111b (6s) == sanity-hsm test 112: State of recorded request ======== 08:50:09 (1748868609) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_112/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled /mnt/lustre/d112.sanity-hsm/f112.sanity-hsm: ARCHIVE waiting (from 0 to EOF) mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 112 (4s) == sanity-hsm test 113: wrong stat after restore ========= 08:50:13 (1748868613) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0195855 s, 53.5 MB/s Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre3 Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_113/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre3"' Waiting 200s for 'SUCCEED' Verifying released state: multiop /mnt/lustre/d113.sanity-hsm/f113.sanity-hsm voO_WRONLY:O_APPEND:_w4c TMPPIPE=/tmp/multiop_open_wait_pipe.7959 File: '/mnt/lustre2/d113.sanity-hsm/f113.sanity-hsm' Size: 1048576 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205306057051 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-06-02 08:50:13.000000000 -0400 Modify: 2025-06-02 08:50:13.000000000 -0400 Change: 2025-06-02 08:50:13.000000000 -0400 Birth: - Stopping client oleg226-client.virtnet /mnt/lustre3 (opts:) PASS 113 (5s) == sanity-hsm test 114: Incompatible request does not set other requests as STARTED ========================================================== 08:50:18 (1748868618) 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00118653 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00113194 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_114/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' PASS 114 (7s) == sanity-hsm test 200: Register/Cancel archive ========== 08:50:25 (1748868625) 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00165819 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_200/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Copytool is suspended on oleg226-client.virtnet Waiting 200s for 'STARTED' Waiting 200s for 'CANCELED' Copytool is continued on oleg226-client.virtnet PASS 200 (5s) == sanity-hsm test 201: Register/Cancel restore ========== 08:50:30 (1748868630) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_201/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 39+0 records in 39+0 records out 40894464 bytes (41 MB) copied, 0.361139 s, 113 MB/s lhsmtool_posix: 1748868631.589844 lhsmtool_posix[7912]: action=1 src=d201.sanity-hsm/f201.sanity-hsm dst=/mnt/lustre/d201.sanity-hsm/f201.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1748868631.592194 lhsmtool_posix[7912]: importing '/mnt/lustre/d201.sanity-hsm/f201.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_201//d201.sanity-hsm/f201.sanity-hsm' lhsmtool_posix: 1748868631.599700 lhsmtool_posix[7912]: imported '/mnt/lustre/d201.sanity-hsm/f201.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_201//0ba0/0x200000bd2:0x72:0x0'=='/tmp/arc1/sanity-hsm.test_201//d201.sanity-hsm/f201.sanity-hsm' lhsmtool_posix: 1748868631.599763 lhsmtool_posix[7912]: process finished, errs: 0 major, 0 minor, rc=0 (Success) mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 201 (5s) == sanity-hsm test 202: Register/Cancel remove =========== 08:50:35 (1748868635) 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00154485 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_202/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Copytool is suspended on oleg226-client.virtnet Waiting 200s for 'STARTED' Waiting 200s for 'CANCELED' PASS 202 (5s) == sanity-hsm test 220A: Changelog for archive =========== 08:50:40 (1748868640) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_220A/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' lustre-MDT0000.3 16HSM 12:50:41.420197720 2025.06.02 0x0 t=[0x200000403:0x163:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 PASS 220A (5s) == sanity-hsm test 220a: Changelog for failed archive ==== 08:50:45 (1748868645) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_220a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl2 cl2' Copytool is suspended on oleg226-client.virtnet Copytool is continued on oleg226-client.virtnet lustre-MDT0000.6 16HSM 12:50:46.560722142 2025.06.02 0x2 t=[0x200000403:0x164:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp p=[0x200000bd2:0x75:0x0] lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0001: Deregistered changelog user #2 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 PASS 220a (5s) == sanity-hsm test 221: Changelog for archive canceled === 08:50:50 (1748868650) 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00165334 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_221/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl3 cl3' Copytool is suspended on oleg226-client.virtnet Waiting 200s for 'CANCELED' Updated after 2s: want 'CANCELED' got 'CANCELED' Copytool is continued on oleg226-client.virtnet lustre-MDT0000.8 16HSM 12:50:53.301585762 2025.06.02 0x7d t=[0x200000403:0x165:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp p=[0x200000bd2:0x75:0x0] lustre-MDT0001: clear the changelog for cl3 of all records lustre-MDT0001: Deregistered changelog user #3 lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0000: Deregistered changelog user #3 PASS 221 (7s) == sanity-hsm test 222a: Changelog for explicit restore == 08:50:57 (1748868657) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_222a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1748868657.793026 lhsmtool_posix[12068]: action=1 src=d222a.sanity-hsm/f222a.sanity-hsm dst=/mnt/lustre/d222a.sanity-hsm/f222a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1748868657.794908 lhsmtool_posix[12068]: importing '/mnt/lustre/d222a.sanity-hsm/f222a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_222a//d222a.sanity-hsm/f222a.sanity-hsm' lhsmtool_posix: 1748868657.803258 lhsmtool_posix[12068]: imported '/mnt/lustre/d222a.sanity-hsm/f222a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_222a//0baa/0x200000bd2:0x78:0x0'=='/tmp/arc1/sanity-hsm.test_222a//d222a.sanity-hsm/f222a.sanity-hsm' lhsmtool_posix: 1748868657.803339 lhsmtool_posix[12068]: process finished, errs: 0 major, 0 minor, rc=0 (Success) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl4 cl4' lustre-MDT0000.11 16HSM 12:50:58.344217300 2025.06.02 0x80 t=[0x200000bd2:0x78:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp p=[0x200000bd2:0x75:0x0] lustre-MDT0001: clear the changelog for cl4 of all records lustre-MDT0001: Deregistered changelog user #4 lustre-MDT0000: clear the changelog for cl4 of all records lustre-MDT0000: Deregistered changelog user #4 PASS 222a (5s) == sanity-hsm test 222b: Changelog for implicit restore == 08:51:02 (1748868662) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_222b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl5 cl5' Waiting 200s for 'SUCCEED' 0e339f9bf886034c50444841bff8eb5d /mnt/lustre/d222b.sanity-hsm/f222b.sanity-hsm lustre-MDT0000.20 16HSM 12:51:05.401014794 2025.06.02 0x80 t=[0x200000403:0x167:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp lustre-MDT0001: clear the changelog for cl5 of all records lustre-MDT0001: Deregistered changelog user #5 lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0000: Deregistered changelog user #5 PASS 222b (7s) == sanity-hsm test 222c: Changelog for failed explicit restore ========================================================== 08:51:09 (1748868669) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_222c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1748868670.129314 lhsmtool_posix[13823]: action=1 src=d222c.sanity-hsm/f222c.sanity-hsm dst=/mnt/lustre/d222c.sanity-hsm/f222c.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1748868670.131647 lhsmtool_posix[13823]: importing '/mnt/lustre/d222c.sanity-hsm/f222c.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_222c//d222c.sanity-hsm/f222c.sanity-hsm' lhsmtool_posix: 1748868670.140014 lhsmtool_posix[13823]: imported '/mnt/lustre/d222c.sanity-hsm/f222c.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_222c//0baf/0x200000bd2:0x7d:0x0'=='/tmp/arc1/sanity-hsm.test_222c//d222c.sanity-hsm/f222c.sanity-hsm' lhsmtool_posix: 1748868670.140086 lhsmtool_posix[13823]: process finished, errs: 0 major, 0 minor, rc=0 (Success) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl6 cl6' Copytool is suspended on oleg226-client.virtnet Waiting 200s for 'STARTED' Copytool is continued on oleg226-client.virtnet lustre-MDT0000.22 16HSM 12:51:12.188933606 2025.06.02 0x82 t=[0x200000bd2:0x7d:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp p=[0x200000bd2:0x7b:0x0] lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0001: Deregistered changelog user #6 lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0000: Deregistered changelog user #6 PASS 222c (7s) == sanity-hsm test 222d: Changelog for failed implicit restore ========================================================== 08:51:16 (1748868676) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_222d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl7 cl7' Waiting 200s for 'SUCCEED' Remove from backend: 0x200000403:0x169:0x0 = /tmp/arc1/sanity-hsm.test_222d/056a/0x200000403:0x169:0x0 md5sum: /mnt/lustre/d222d.sanity-hsm/f222d.sanity-hsm: No data available lustre-MDT0000.29 16HSM 12:51:19.241108234 2025.06.02 0x82 t=[0x200000403:0x169:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp lustre-MDT0001: clear the changelog for cl7 of all records lustre-MDT0001: Deregistered changelog user #7 lustre-MDT0000: clear the changelog for cl7 of all records lustre-MDT0000: Deregistered changelog user #7 PASS 222d (7s) == sanity-hsm test 223a: Changelog for restore canceled (import case) ========================================================== 08:51:23 (1748868683) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_223a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' 39+0 records in 39+0 records out 40894464 bytes (41 MB) copied, 0.419126 s, 97.6 MB/s mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl8 cl8' lhsmtool_posix: 1748868686.322882 lhsmtool_posix[15701]: action=1 src=d223a.sanity-hsm/f223a.sanity-hsm dst=/mnt/lustre/d223a.sanity-hsm/f223a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1748868686.325484 lhsmtool_posix[15701]: importing '/mnt/lustre/d223a.sanity-hsm/f223a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_223a//d223a.sanity-hsm/f223a.sanity-hsm' lhsmtool_posix: 1748868686.338763 lhsmtool_posix[15701]: imported '/mnt/lustre/d223a.sanity-hsm/f223a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_223a//0b53/0x200000bd2:0x81:0x0'=='/tmp/arc1/sanity-hsm.test_223a//d223a.sanity-hsm/f223a.sanity-hsm' lhsmtool_posix: 1748868686.338843 lhsmtool_posix[15701]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Waiting 200s for 'STARTED' Waiting 200s for 'CANCELED' Updated after 2s: want 'CANCELED' got 'CANCELED' Waiting 200s for 'SUCCEED' Waiting 190s for 'SUCCEED' Waiting 180s for 'SUCCEED' Updated after 29s: want 'SUCCEED' got 'SUCCEED' lustre-MDT0000.36 16HSM 12:51:56.525903983 2025.06.02 0xfd t=[0x200000bd2:0x81:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp lustre-MDT0001: clear the changelog for cl8 of all records lustre-MDT0001: Deregistered changelog user #8 lustre-MDT0000: clear the changelog for cl8 of all records lustre-MDT0000: Deregistered changelog user #8 PASS 223a (38s) == sanity-hsm test 223b: Changelog for restore canceled (release case) ========================================================== 08:52:01 (1748868721) 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00158615 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_223b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl9 cl9' Waiting 200s for 'SUCCEED' Copytool is suspended on oleg226-client.virtnet Waiting 200s for 'STARTED' Waiting 200s for 'CANCELED' Updated after 2s: want 'CANCELED' got 'CANCELED' Copytool is continued on oleg226-client.virtnet lustre-MDT0000.43 16HSM 12:52:07.333403054 2025.06.02 0xfd t=[0x200000403:0x16d:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp p=[0x200000bd2:0x7b:0x0] lustre-MDT0001: clear the changelog for cl9 of all records lustre-MDT0001: Deregistered changelog user #9 lustre-MDT0000: clear the changelog for cl9 of all records lustre-MDT0000: Deregistered changelog user #9 PASS 223b (10s) == sanity-hsm test 224A: Changelog for remove ============ 08:52:11 (1748868731) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_224A/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl10 cl10' Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' lustre-MDT0000.48 16HSM 12:52:14.362720204 2025.06.02 0x200 t=[0x200000403:0x16f:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp lustre-MDT0001: clear the changelog for cl10 of all records lustre-MDT0001: Deregistered changelog user #10 lustre-MDT0000: clear the changelog for cl10 of all records lustre-MDT0000: Deregistered changelog user #10 PASS 224A (8s) == sanity-hsm test 224a: Changelog for failed remove ===== 08:52:19 (1748868739) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_224a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl11 cl11' Waiting 200s for 'SUCCEED' Remove from backend: 0x200000403:0x170:0x0 = /tmp/arc1/sanity-hsm.test_224a/0573/0x200000403:0x170:0x0 Copytool is suspended on oleg226-client.virtnet Waiting 200s for 'STARTED' Copytool is continued on oleg226-client.virtnet lustre-MDT0000.53 16HSM 12:52:23.628332768 2025.06.02 0x202 t=[0x200000403:0x170:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp p=[0x200000bd2:0x85:0x0] lustre-MDT0001: clear the changelog for cl11 of all records lustre-MDT0001: Deregistered changelog user #11 lustre-MDT0000: clear the changelog for cl11 of all records lustre-MDT0000: Deregistered changelog user #11 PASS 224a (8s) == sanity-hsm test 225: Changelog for remove canceled ==== 08:52:27 (1748868747) Test disabled PASS 225 (2s) == sanity-hsm test 226: changelog for last rm/mv with exiting archive ========================================================== 08:52:29 (1748868749) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_226/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 0x200000403:0x173:0x0 mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl12 cl12' Waiting 200s for 'SUCCEED' lustre-MDT0000.54 16HSM 12:52:30.628883001 2025.06.02 0x280 t=[0x200000403:0x171:0x0] ef=0x13 u=0:0 nid=0@<0:0> lustre-MDT0000.55 16HSM 12:52:30.645796578 2025.06.02 0x280 t=[0x200000403:0x171:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp lustre-MDT0000.56 16HSM 12:52:30.645866696 2025.06.02 0x0 t=[0x200000403:0x171:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp lustre-MDT0000.57 16HSM 12:52:31.646011351 2025.06.02 0x280 t=[0x200000403:0x172:0x0] ef=0x13 u=0:0 nid=0@<0:0> lustre-MDT0000.58 16HSM 12:52:31.667324745 2025.06.02 0x280 t=[0x200000403:0x172:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp lustre-MDT0000.59 16HSM 12:52:31.667391238 2025.06.02 0x0 t=[0x200000403:0x172:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp lustre-MDT0000.60 06UNLNK 12:52:31.776233910 2025.06.02 0x3 t=[0x200000403:0x171:0x0] j=rm.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp p=[0x200000bd2:0x86:0x0] f226.sanity-hsm-1 lustre-MDT0000.60 06UNLNK 12:52:31.776233910 2025.06.02 0x3 t=[0x200000403:0x171:0x0] j=rm.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp p=[0x200000bd2:0x86:0x0] f226.sanity-hsm-1 lustre-MDT0000.61 08RENME 12:52:31.883763307 2025.06.02 0x3 t=[0x200000403:0x172:0x0] j=mv.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp p=[0x200000bd2:0x86:0x0] f226.sanity-hsm-2 s=[0x200000403:0x173:0x0] sp=[0x200000bd2:0x86:0x0] f226.sanity-hsm-3 lustre-MDT0001: clear the changelog for cl12 of all records lustre-MDT0001: Deregistered changelog user #12 lustre-MDT0000: clear the changelog for cl12 of all records lustre-MDT0000: Deregistered changelog user #12 PASS 226 (6s) == sanity-hsm test 227: changelog when explicit setting of HSM flags ========================================================== 08:52:35 (1748868755) 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00146831 s, 0.0 kB/s mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl13 cl13' lustre-MDT0001: clear the changelog for cl13 of all records lustre-MDT0001: Deregistered changelog user #13 lustre-MDT0000: clear the changelog for cl13 of all records lustre-MDT0000: Deregistered changelog user #13 PASS 227 (6s) == sanity-hsm test 228: On released file, return extend to FIEMAP. For [cp,tar] --sparse ========================================================== 08:52:41 (1748868761) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_228/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0188701 s, 55.6 MB/s Waiting 200s for 'SUCCEED' /mnt/lustre/f228.sanity-hsm: 1 extent found tar: Removing leading `/' from member names mnt/lustre/f228.sanity-hsm PASS 228 (5s) == sanity-hsm test 250: Coordinator max request ========== 08:52:46 (1748868766) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_250/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00108124 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00136379 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00124636 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00134264 s, 0.0 kB/s Waiting 200s for 'SUCCEED' 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.000862838 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00115496 s, 0.0 kB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Waiting 200s for 'SUCCEED' Copytool is suspended on oleg226-client.virtnet Waiting 200s for 'STARTED' Updated after 2s: want 'STARTED' got 'STARTED' Waiting 200s for 'STARTED' Waiting 200s for 'STARTED' Updated after 2s: want 'STARTED' got 'STARTED' Copytool is continued on oleg226-client.virtnet PASS 250 (14s) == sanity-hsm test 251: Coordinator request timeout ====== 08:53:00 (1748868780) 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0011375 s, 0.0 kB/s mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_251/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Copytool is suspended on oleg226-client.virtnet Waiting 200s for 'STARTED' Waiting 200s for 'CANCELED' PASS 251 (9s) == sanity-hsm test 252: Timeout'ed running archive of a removed file should be canceled ========================================================== 08:53:09 (1748868789) 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.000961319 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_252/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Copytool is suspended on oleg226-client.virtnet Waiting 200s for 'STARTED' Waiting 200s for 'CANCELED' Copytool is continued on oleg226-client.virtnet PASS 252 (8s) == sanity-hsm test 253: Check for wrong file size after release ========================================================== 08:53:17 (1748868797) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_253/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 10+0 records in 10+0 records out 10000000 bytes (10 MB) copied, 0.0717117 s, 139 MB/s fail_loc=0x807 Cannot send HSM request (use of /mnt/lustre/d253.sanity-hsm/f253.sanity-hsm): Invalid argument could not release file PASS 253 (2s) == sanity-hsm test 254a: Request counters are initialized to zero ========================================================== 08:53:19 (1748868799) PASS 254a (3s) == sanity-hsm test 254b: Request counters are correctly incremented and decremented ========================================================== 08:53:22 (1748868802) Will launch 37 requests of each type Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_254b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Checking archive requests Copytool is suspended on oleg226-client.virtnet Copytool is continued on oleg226-client.virtnet Checking restore requests Copytool is suspended on oleg226-client.virtnet Copytool is continued on oleg226-client.virtnet Checking remove requests Copytool is suspended on oleg226-client.virtnet Copytool is continued on oleg226-client.virtnet PASS 254b (11s) == sanity-hsm test 255: Copytool registration wakes the coordinator up ========================================================== 08:53:33 (1748868813) 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00172764 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_255/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Killing existing copytools on oleg226-client.virtnet copytools stopped in 0s 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00123339 s, 0.0 kB/s 0x200000403:0x1cf:0x0 mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled Waiting 200s for '' Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_255/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' PASS 255 (11s) == sanity-hsm test 260a: Restore request have priority over other requests ========================================================== 08:53:44 (1748868824) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0364604 s, 28.8 MB/s 0x200000403:0x1d0:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0360894 s, 29.1 MB/s 0x200000403:0x1d1:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0367187 s, 28.6 MB/s 0x200000403:0x1d2:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0352201 s, 29.8 MB/s 0x200000403:0x1d3:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0376161 s, 27.9 MB/s 0x200000403:0x1d4:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0319959 s, 32.8 MB/s 0x200000403:0x1d5:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0392683 s, 26.7 MB/s 0x200000403:0x1d6:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0311802 s, 33.6 MB/s 0x200000403:0x1d7:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0365902 s, 28.7 MB/s 0x200000403:0x1d8:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.030432 s, 34.5 MB/s 0x200000403:0x1d9:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0395061 s, 26.5 MB/s 0x200000403:0x1da:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0309182 s, 33.9 MB/s 0x200000403:0x1db:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0350955 s, 29.9 MB/s 0x200000403:0x1dc:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0308733 s, 34.0 MB/s 0x200000403:0x1dd:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0375261 s, 27.9 MB/s 0x200000403:0x1de:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0325308 s, 32.2 MB/s 0x200000403:0x1df:0x0 Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Killing existing copytools on oleg226-client.virtnet copytools stopped in 0s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' ARCHIVE ARCHIVE RESTORE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE PASS 260a (11s) == sanity-hsm test 260b: Restore request have priority over other requests ========================================================== 08:53:56 (1748868836) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0355027 s, 29.5 MB/s 0x200000403:0x1e2:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0293084 s, 35.8 MB/s 0x200000403:0x1e3:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0323662 s, 32.4 MB/s 0x200000403:0x1e4:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0317411 s, 33.0 MB/s 0x200000403:0x1e5:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0367266 s, 28.6 MB/s 0x200000403:0x1e6:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0331114 s, 31.7 MB/s 0x200000403:0x1e7:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0290279 s, 36.1 MB/s 0x200000403:0x1e8:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0279507 s, 37.5 MB/s 0x200000403:0x1e9:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.029051 s, 36.1 MB/s 0x200000403:0x1ea:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0291182 s, 36.0 MB/s 0x200000403:0x1eb:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0262876 s, 39.9 MB/s 0x200000403:0x1ec:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0256922 s, 40.8 MB/s 0x200000403:0x1ed:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0329753 s, 31.8 MB/s 0x200000403:0x1ee:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0321759 s, 32.6 MB/s 0x200000403:0x1ef:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0343127 s, 30.6 MB/s 0x200000403:0x1f0:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0283706 s, 37.0 MB/s 0x200000403:0x1f1:0x0 Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2 "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Killing existing copytools on oleg226-client.virtnet copytools stopped in 0s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2 "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' RESTORE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE PASS 260b (13s) == sanity-hsm test 260c: Requests are not reordered on the 'hot' path of the coordinator ========================================================== 08:54:09 (1748868849) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.035953 s, 29.2 MB/s 0x200000403:0x1f3:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0281555 s, 37.2 MB/s 0x200000403:0x1f4:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0309089 s, 33.9 MB/s 0x200000403:0x1f5:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0315246 s, 33.3 MB/s 0x200000403:0x1f6:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0324545 s, 32.3 MB/s 0x200000403:0x1f7:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0270081 s, 38.8 MB/s 0x200000403:0x1f8:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0314954 s, 33.3 MB/s 0x200000403:0x1f9:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0375933 s, 27.9 MB/s 0x200000403:0x1fa:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0364968 s, 28.7 MB/s 0x200000403:0x1fb:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0330744 s, 31.7 MB/s 0x200000403:0x1fc:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0304566 s, 34.4 MB/s 0x200000403:0x1fd:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0300152 s, 34.9 MB/s 0x200000403:0x1fe:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0259606 s, 40.4 MB/s 0x200000403:0x1ff:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0333709 s, 31.4 MB/s 0x200000403:0x200:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0379183 s, 27.7 MB/s 0x200000403:0x201:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0315315 s, 33.3 MB/s 0x200000403:0x202:0x0 Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2 "/mnt/lustre2"' Killing existing copytools on oleg226-client.virtnet copytools stopped in 0s mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2 "/mnt/lustre2"' Waiting 200s for 'SUCCEED' ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE RESTORE PASS 260c (13s) == sanity-hsm test 261: Report 0 bytes size after HSM release ========================================================== 08:54:22 (1748868862) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_261/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00210639 s, 3.9 MB/s Waiting 200s for 'SUCCEED' /mnt/lustre/d261.sanity-hsm/f261.sanity-hsm: (0x00000009) exists archived, archive_id:2 Waiting 200s for 'SUCCEED' /mnt/lustre/d261.sanity-hsm/f261.sanity-hsm: (0x0000000d) released exists archived, archive_id:2 PASS 261 (5s) == sanity-hsm test 262: The client should return 1 block for HSM released files ========================================================== 08:54:27 (1748868867) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_262/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00409875 s, 2.0 MB/s PASS 262 (3s) == sanity-hsm test 300: On disk coordinator state kept between MDT umount/mount ========================================================== 08:54:30 (1748868870) Stop coordinator and remove coordinator state at mount mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown Failing mds1 on oleg226-server Stopping /mnt/lustre-mds1 (opts:) on oleg226-server 08:54:39 (1748868879) shut down facet: mds1 facet_host: oleg226-server facet_failover_host: oleg226-server Failover mds1 to oleg226-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0000 08:54:52 (1748868892) targets are mounted 08:54:52 (1748868892) facet_failover done oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Set coordinator start at mount, and start coordinator Failing mds1 on oleg226-server Stopping /mnt/lustre-mds1 (opts:) on oleg226-server 08:55:21 (1748868921) shut down facet: mds1 facet_host: oleg226-server facet_failover_host: oleg226-server Failover mds1 to oleg226-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0000 08:55:34 (1748868934) targets are mounted 08:55:34 (1748868934) facet_failover done oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 300 (72s) == sanity-hsm test 301: HSM tunnable are persistent ====== 08:55:42 (1748868942) Failing mds1 on oleg226-server Stopping /mnt/lustre-mds1 (opts:) on oleg226-server 08:55:44 (1748868944) shut down facet: mds1 facet_host: oleg226-server facet_failover_host: oleg226-server Failover mds1 to oleg226-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0000 08:55:58 (1748868958) targets are mounted 08:55:58 (1748868958) facet_failover done oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 301 (24s) == sanity-hsm test 302: HSM tunnable are persistent when CDT is off ========================================================== 08:56:06 (1748868966) mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown Failing mds1 on oleg226-server Stopping /mnt/lustre-mds1 (opts:) on oleg226-server 08:56:09 (1748868969) shut down facet: mds1 facet_host: oleg226-server facet_failover_host: oleg226-server Failover mds1 to oleg226-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0000 08:56:22 (1748868982) targets are mounted 08:56:22 (1748868982) facet_failover done oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg226-server Stopping /mnt/lustre-mds2 (opts:) on oleg226-server 08:56:29 (1748868989) shut down facet: mds2 facet_host: oleg226-server facet_failover_host: oleg226-server Failover mds2 to oleg226-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0001 08:56:43 (1748869003) targets are mounted 08:56:43 (1748869003) facet_failover done oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 302 (45s) == sanity-hsm test 400: Single request is sent to the right MDT ========================================================== 08:56:51 (1748869011) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_400/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0353052 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0299694 s, 35.0 MB/s archive successful on mdt0 archive successful on mdt1 PASS 400 (2s) == sanity-hsm test 401: Compound requests split and sent to their respective MDTs ========================================================== 08:56:53 (1748869013) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_401/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0271103 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0275244 s, 38.1 MB/s Waiting 200s for 'SUCCEED' archive successful on mdt0 archive successful on mdt1 PASS 401 (4s) == sanity-hsm test 402a: Copytool start fails if all MDTs are inactive ========================================================== 08:56:57 (1748869017) deactivate lustre-MDT0000-mdc-ffff8800a514a000 on oleg226-client.virtnet deactivate lustre-MDT0001-mdc-ffff8800a514a000 on oleg226-client.virtnet deactivate lustre-MDT0000-mdc-ffff8800b6d11000 on oleg226-client.virtnet deactivate lustre-MDT0001-mdc-ffff8800b6d11000 on oleg226-client.virtnet Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_402a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Failed to start copytool agt1 on 'oleg226-client.virtnet' pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 uuid not found in agent list on mds1 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 uuid not found in agent list on mds2 activate lustre-MDT0000-mdc-ffff8800a514a000 on oleg226-client.virtnet activate lustre-MDT0001-mdc-ffff8800a514a000 on oleg226-client.virtnet activate lustre-MDT0000-mdc-ffff8800b6d11000 on oleg226-client.virtnet activate lustre-MDT0001-mdc-ffff8800b6d11000 on oleg226-client.virtnet PASS 402a (2s) == sanity-hsm test 402b: CDT must retry request upon slow start of CT ========================================================== 08:57:00 (1748869020) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_402b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' fail_loc=0x14d fail_loc=0 Waiting 200s for 'SUCCEED' Updated after 3s: want 'SUCCEED' got 'SUCCEED' PASS 402b (14s) == sanity-hsm test 403: Copytool starts with inactive MDT and register on reconnect ========================================================== 08:57:14 (1748869034) deactivate lustre-MDT0001-mdc-ffff8800a514a000 on oleg226-client.virtnet deactivate lustre-MDT0001-mdc-ffff8800b6d11000 on oleg226-client.virtnet Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_403/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' oleg226-client.virtnet: executing pgrep --pidfile=/var/run/lhsmtool_posix.pid --list-full hsmtool oleg226-client.virtnet: executing get_client_uuid /mnt/lustre2 found agent uuid=c8af5ef9-f4c9-4606-93aa-65a65d478e84 archive_id=ANY requests=[current:0 ok:0 errors:0] on mds1 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 uuid not found in agent list on mds2 oleg226-client.virtnet: 11533 activate lustre-MDT0001-mdc-ffff8800a514a000 on oleg226-client.virtnet activate lustre-MDT0001-mdc-ffff8800b6d11000 on oleg226-client.virtnet found agent uuid=c8af5ef9-f4c9-4606-93aa-65a65d478e84 archive_id=ANY requests=[current:0 ok:0 errors:0] on mds1 found agent uuid=c8af5ef9-f4c9-4606-93aa-65a65d478e84 archive_id=ANY requests=[current:0 ok:0 errors:0] on mds2 PASS 403 (5s) == sanity-hsm test 404: Inactive MDT does not block requests for active MDTs ========================================================== 08:57:19 (1748869039) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_404/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.029328 s, 35.8 MB/s deactivate lustre-MDT0001-mdc-ffff8800a514a000 on oleg226-client.virtnet deactivate lustre-MDT0001-mdc-ffff8800b6d11000 on oleg226-client.virtnet Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' archive successful on mdt0 activate lustre-MDT0001-mdc-ffff8800a514a000 on oleg226-client.virtnet activate lustre-MDT0001-mdc-ffff8800b6d11000 on oleg226-client.virtnet PASS 404 (3s) == sanity-hsm test 405: archive and release under striped directory ========================================================== 08:57:22 (1748869042) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_405/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0185864 s, 56.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0186497 s, 56.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0216267 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0176291 s, 59.5 MB/s Waiting 200s for 'SUCCEED' archive successful on 0x240000bd2:0x1:0x0 archive successful on 0x240000bd2:0x2:0x0 archive successful on 0x240000bd2:0x3:0x0 archive successful on 0x2000032e3:0x6:0x0 PASS 405 (7s) == sanity-hsm test 406: attempting to migrate HSM archived files is safe ========================================================== 08:57:30 (1748869050) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0350851 s, 29.9 MB/s old fid 0x2000032e3:0x9:0x0 Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_406/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' lfs migrate: /mnt/lustre/d406.sanity-hsm/f406.sanity-hsm migrate failed: Operation not supported (95) Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Waiting 200s for 'SUCCEED' new fid 0x240000bd3:0x7:0x0 debug=+trace Waiting 200s for 'SUCCEED' PASS 406 (9s) == sanity-hsm test 407: Check for double RESTORE records in llog ========================================================== 08:57:39 (1748869059) 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00130245 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_407/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' fail_val=5 fail_loc=0x164 Copytool is suspended on oleg226-client.virtnet mdt.lustre-MDT0000.hsm.actions= lrh=[type=10680000 len=136 idx=6/1] fid=[0x200000403:0x209:0x0] dfid=[0x200000403:0x209:0x0] compound/cookie=0x0/0x683d9f76 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/2] fid=[0x200000403:0x20d:0x0] dfid=[0x200000403:0x20d:0x0] compound/cookie=0x0/0x683d9f77 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/3] fid=[0x2000032e2:0x2:0x0] dfid=[0x2000032e2:0x2:0x0] compound/cookie=0x0/0x683d9f78 action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/4] fid=[0x2000032e3:0x3:0x0] dfid=[0x2000032e3:0x3:0x0] compound/cookie=0x0/0x683d9f79 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/5] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x6:0x0] compound/cookie=0x0/0x683d9f7a action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/6] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x6:0x0] compound/cookie=0x0/0x683d9f7b action=RESTORE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/7] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x683d9f7c action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/8] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x683d9f7d action=RESTORE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/9] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x683d9f7e action=REMOVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/10] fid=[0x2000032e3:0xc:0x0] dfid=[0x2000032e3:0xc:0x0] compound/cookie=0x0/0x683d9f7f action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] Failing mds1 on oleg226-server Stopping /mnt/lustre-mds1 (opts:) on oleg226-server 08:57:47 (1748869067) shut down facet: mds1 facet_host: oleg226-server facet_failover_host: oleg226-server Failover mds1 to oleg226-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0000 08:58:00 (1748869080) targets are mounted 08:58:00 (1748869080) facet_failover done oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 mdt.lustre-MDT0000.hsm.actions= lrh=[type=10680000 len=136 idx=6/3] fid=[0x2000032e2:0x2:0x0] dfid=[0x2000032e2:0x2:0x0] compound/cookie=0x0/0x683d9f78 action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/4] fid=[0x2000032e3:0x3:0x0] dfid=[0x2000032e3:0x3:0x0] compound/cookie=0x0/0x683d9f79 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/5] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x6:0x0] compound/cookie=0x0/0x683d9f7a action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/6] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x6:0x0] compound/cookie=0x0/0x683d9f7b action=RESTORE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/7] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x683d9f7c action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/8] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x683d9f7d action=RESTORE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/9] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x683d9f7e action=REMOVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/10] fid=[0x2000032e3:0xc:0x0] dfid=[0x2000032e3:0xc:0x0] compound/cookie=0x0/0x683d9f7f action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/11] fid=[0x2000032e3:0xc:0x0] dfid=[0x2000032e3:0xc:0x0] compound/cookie=0x0/0x683d9f80 action=RESTORE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[] Copytool is continued on oleg226-client.virtnet d41d8cd98f00b204e9800998ecf8427e /mnt/lustre/d407.sanity-hsm/f407.sanity-hsm d41d8cd98f00b204e9800998ecf8427e /mnt/lustre2/d407.sanity-hsm/f407.sanity-hsm pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 mdt.lustre-MDT0000.hsm.actions= lrh=[type=10680000 len=136 idx=6/3] fid=[0x2000032e2:0x2:0x0] dfid=[0x2000032e2:0x2:0x0] compound/cookie=0x0/0x683d9f78 action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/4] fid=[0x2000032e3:0x3:0x0] dfid=[0x2000032e3:0x3:0x0] compound/cookie=0x0/0x683d9f79 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/5] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x6:0x0] compound/cookie=0x0/0x683d9f7a action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/6] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x6:0x0] compound/cookie=0x0/0x683d9f7b action=RESTORE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/7] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x683d9f7c action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/8] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x683d9f7d action=RESTORE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/9] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x683d9f7e action=REMOVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/10] fid=[0x2000032e3:0xc:0x0] dfid=[0x2000032e3:0xc:0x0] compound/cookie=0x0/0x683d9f7f action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/11] fid=[0x2000032e3:0xc:0x0] dfid=[0x2000032e3:0xc:0x0] compound/cookie=0x0/0x683d9f80 action=RESTORE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] PASS 407 (35s) == sanity-hsm test 408: Verify fiemap on release file ==== 08:58:14 (1748869094) Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_408/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.0165672 s, 4.0 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.0126098 s, 5.2 MB/s File: '/mnt/lustre/f408.sanity-hsm' Size: 262144 Blocks: 256 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115406615871500 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-06-02 08:58:14.000000000 -0400 Modify: 2025-06-02 08:58:14.000000000 -0400 Change: 2025-06-02 08:58:14.000000000 -0400 Birth: - disk usage: 131072 /mnt/lustre/f408.sanity-hsm file size: 262144 /mnt/lustre/f408.sanity-hsm extent 0 in offset 0, length 262144 flags (0x3): LAST UNKNOWN No unwritten extents, extents number 1, sum of lengths 262144, expected sum 262144 PASS 408 (2s) == sanity-hsm test 409a: Coordinator should not stop when in use ========================================================== 08:58:16 (1748869096) 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.002413 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_409a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' fail_val=5 fail_loc=0x164 cat: /mnt/lustre/d409a.sanity-hsm/f409a.sanity-hsm: No data available mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled Killing existing copytools on oleg226-client.virtnet copytools stopped in 0s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_409a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 409a (12s) == sanity-hsm test 409b: getattr released file with CDT stopped after remount ========================================================== 08:58:29 (1748869109) 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00140084 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_409b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Killing existing copytools on oleg226-client.virtnet copytools stopped in 0s Failing mds1 on oleg226-server Stopping /mnt/lustre-mds1 (opts:) on oleg226-server 08:58:51 (1748869131) shut down facet: mds1 facet_host: oleg226-server facet_failover_host: oleg226-server Failover mds1 to oleg226-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0000 08:59:04 (1748869144) targets are mounted 08:59:04 (1748869144) facet_failover done oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec File: '/mnt/lustre/d409b.sanity-hsm/f409b.sanity-hsm' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115406632648720 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-06-02 08:58:29.000000000 -0400 Modify: 2025-06-02 08:58:29.000000000 -0400 Change: 2025-06-02 08:58:29.000000000 -0400 Birth: - PASS 409b (63s) == sanity-hsm test 410: lfs data_version -s allows release of force-archived file ========================================================== 08:59:33 (1748869173) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.059561 s, 17.6 MB/s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_410/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Cannot send HSM request (use of /mnt/lustre/d410.sanity-hsm/f410.sanity-hsm): Operation not permitted PASS 410 (2s) == sanity-hsm test 411: hsm_ops rbac role ================ 08:59:35 (1748869175) On MGS 192.168.202.126, active = nodemap.active=1 waiting 10 secs for sync On MGS 192.168.202.126, test_411.trusted_nodemap = nodemap.test_411.trusted_nodemap=1 waiting 10 secs for sync Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_411/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' /mnt/lustre/d411.sanity-hsm/fileA: (0x00000000) Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Waiting 200s for 'SUCCEED' Killing existing copytools on oleg226-client.virtnet copytools stopped in 0s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_411/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' On MGS 192.168.202.126, test_411.rbac = nodemap.test_411.rbac=file_perms,dne_ops,quota_ops,byfid_ops,chlg_ops,fscrypt_admin,server_upcall,ignore_root_prjquota,local_admin waiting 10 secs for sync /mnt/lustre/d411.sanity-hsm/fileB: (0x00000000) Cannot send HSM request (use of /mnt/lustre/d411.sanity-hsm/fileB): Permission denied Cannot send HSM request (use of /mnt/lustre/d411.sanity-hsm/fileB): Operation not permitted Cannot send HSM request (use of /mnt/lustre/d411.sanity-hsm/fileB): Permission denied Cannot send HSM request (use of /mnt/lustre/d411.sanity-hsm/fileB): Permission denied lfs hsm_set: change hsm flags for '/mnt/lustre/d411.sanity-hsm/fileB' failed: Permission denied lfs hsm_clear: change hsm flags for '/mnt/lustre/d411.sanity-hsm/fileB' failed: Permission denied On MGS 192.168.202.126, active = nodemap.active=0 waiting 10 secs for sync oleg226-server: error: nodemap 'test_411' does not exist oleg226-server: nodemap_del: Invalid argument pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 On MGS 192.168.202.126, active = nodemap.active=0 waiting 10 secs for sync PASS 411 (63s) == sanity-hsm test 500: various LLAPI HSM tests ========== 09:00:38 (1748869238) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d500.sanity-hsm test1 @1748869254: Register/unregister copytool 2000x to check for leaks ...... pass test2 @1748869254: Re/un-register copytool multiple times without error ....... pass test3 @1748869254: Pass bad parameters to llapi_hsm_copytool_register() ....... pass test4 @1748869254: Bad parameters to llapi_hsm_copytool_unregister() .......... pass test5 @1748869254: Test llapi_hsm_copytool_recv() in non-blocking mode ........ pass test6 @1748869254: Test llapi_hsm_copytool_recv() with bogus parameters ....... pass test7 @1748869254: Test event polling (without actual traffic) ................ pass test50 @1748869254: Test llapi_hsm_state_get()/get_fd() ....................... pass test51 @1748869254: Test llapi_hsm_state_set_fd() ............................. pass test52 @1748869254: Test llapi_hsm_current_action() ........................... pass test100 @1748869255: Simple archive creation with no progress reported. ....... pass test101 @1748869256: Simple archive creation with progress every byte. ........ pass test102 @1748869258: Archive creation with progress every byte, backwards ..... pass test103 @1748869259: Archive creation with a single progress report ........... pass test104 @1748869260: Archive creation with two progress reports ............... pass test105 @1748869261: Archive creation with one bogus progress report .......... pass test106 @1748869262: Archive creation with one empty progress report .......... pass test107 @1748869263: Archive creation with one bogus progress report .......... pass test108 @1748869265: Archive creation with same progress report each time ..... pass test109 @1748869266: Archive creation with one report, with large number ...... pass test110 @1748869267: Archive with 10 progress reports, checking progress ...... pass test111 @1748869268: Archive with 10 reports in reverse, checking progress .... pass test112 @1748869269: Archive with 10 reports, duplicated, check progress ...... pass test113 @1748869270: Archive with 9 reports, with 20% overlapping coverage .... pass PASS 500 (34s) == sanity-hsm test 600: Changelog fields 'u=' and 'nid=' ========================================================== 09:01:12 (1748869272) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d600.sanity-hsm/f600.sanity-hsm] Got NID '192.168.202.26@tcp' lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 PASS 600 (4s) == sanity-hsm test 601: OPEN Changelog entry ============= 09:01:16 (1748869276) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl2 cl2' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0000.4 10OPEN 13:01:18.146579505 2025.06.02 0x1 t=[0x2000032e2:0x1b:0x0] j=cat.0 ef=0x17 u=0:0 nid=192.168.202.26@tcp m=r-- p=[0x2000032e2:0x1a:0x0] lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0001: Deregistered changelog user #2 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 PASS 601 (5s) == sanity-hsm test 602: Changelog record CLOSE only if open+write or OPEN recorded ========================================================== 09:01:21 (1748869281) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl3 cl3' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0001: clear the changelog for cl3 of all records lustre-MDT0001.9 11CLOSE 13:01:23.257088582 2025.06.02 0x1 t=[0x240000bd3:0x1d:0x0] j=cat.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0001: clear the changelog for cl3 of all records lustre-MDT0001.10 10OPEN 13:01:23.353452082 2025.06.02 0x242 t=[0x240000bd3:0x1d:0x0] j=bash.0 ef=0x17 u=0:0 nid=192.168.202.26@tcp m=-w- p=[0x240000bd3:0x1c:0x0] lustre-MDT0001.11 23GXATR 13:01:23.355076913 2025.06.02 0x0 t=[0x240000bd3:0x1d:0x0] j=bash.0 ef=0x1b u=0:0 nid=192.168.202.26@tcp x=user.job lustre-MDT0001.12 13TRUNC 13:01:23.356433328 2025.06.02 0xe t=[0x240000bd3:0x1d:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp p=[0x240000bd3:0x1c:0x0] lustre-MDT0001.13 11CLOSE 13:01:23.362354317 2025.06.02 0x242 t=[0x240000bd3:0x1d:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp lustre-MDT0001.13 11CLOSE 13:01:23.362354317 2025.06.02 0x242 t=[0x240000bd3:0x1d:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp mdd.lustre-MDT0000.changelog_mask=-OPEN mdd.lustre-MDT0001.changelog_mask=-OPEN lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0001: clear the changelog for cl3 of all records f lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0001: clear the changelog for cl3 of all records lustre-MDT0001.14 13TRUNC 13:01:23.928248701 2025.06.02 0xe t=[0x240000bd3:0x1d:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp p=[0x240000bd3:0x1c:0x0] lustre-MDT0001.15 11CLOSE 13:01:23.931578050 2025.06.02 0x242 t=[0x240000bd3:0x1d:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp lustre-MDT0001.15 11CLOSE 13:01:23.931578050 2025.06.02 0x242 t=[0x240000bd3:0x1d:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp lustre-MDT0001: clear the changelog for cl3 of all records lustre-MDT0001: Deregistered changelog user #3 lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0000: Deregistered changelog user #3 PASS 602 (6s) == sanity-hsm test 603: GETXATTR Changelog entry ========= 09:01:27 (1748869287) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl4 cl4' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL lustre-MDT0000: clear the changelog for cl4 of all records lustre-MDT0001: clear the changelog for cl4 of all records getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/d603.sanity-hsm/f603.sanity-hsm user.xattr1="value1" lustre-MDT0000.10 23GXATR 13:01:29.020957080 2025.06.02 0x0 t=[0x2000032e2:0x1d:0x0] j=getfattr.0 ef=0x1b u=0:0 nid=192.168.202.26@tcp x=user.xattr1 lustre-MDT0001: clear the changelog for cl4 of all records lustre-MDT0001: Deregistered changelog user #4 lustre-MDT0000: clear the changelog for cl4 of all records lustre-MDT0000: Deregistered changelog user #4 PASS 603 (5s) == sanity-hsm test 604: NOPEN Changelog entry ============ 09:01:32 (1748869292) mdd.lustre-MDT0000.changelog_deniednext=20 mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl5 cl5' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0001: clear the changelog for cl5 of all records running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cat] [/mnt/lustre2/d604.sanity-hsm/f604.sanity-hsm] cat: /mnt/lustre2/d604.sanity-hsm/f604.sanity-hsm: Permission denied lustre-MDT0000.15 24NOPEN 13:01:34.213798900 2025.06.02 0x1 t=[0x2000032e2:0x1f:0x0] j=cat.500 ef=0x17 u=500:500 nid=192.168.202.26@tcp m=r-- Got NID '192.168.202.26@tcp' lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0001: clear the changelog for cl5 of all records running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cat] [/mnt/lustre2/d604.sanity-hsm/f604.sanity-hsm] cat: /mnt/lustre2/d604.sanity-hsm/f604.sanity-hsm: Permission denied lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0001: clear the changelog for cl5 of all records running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cat] [/mnt/lustre2/d604.sanity-hsm/f604.sanity-hsm] cat: /mnt/lustre2/d604.sanity-hsm/f604.sanity-hsm: Permission denied lustre-MDT0000.16 24NOPEN 13:01:54.446482413 2025.06.02 0x1 t=[0x2000032e2:0x1f:0x0] j=cat.500 ef=0x17 u=500:500 nid=192.168.202.26@tcp m=r-- p=[0x240000bd3:0x1c:0x0] Got NID '192.168.202.26@tcp' lustre-MDT0001: clear the changelog for cl5 of all records lustre-MDT0001: Deregistered changelog user #5 lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0000: Deregistered changelog user #5 mdd.lustre-MDT0000.changelog_deniednext=60 PASS 604 (27s) == sanity-hsm test 605: Test OPEN and CLOSE rate limit in Changelogs ========================================================== 09:02:00 (1748869320) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl6 cl6' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0001.19 10OPEN 13:02:01.675837453 2025.06.02 0x43 t=[0x240000bd3:0x1f:0x0] j=bash.0 ef=0x17 u=0:0 nid=192.168.202.26@tcp m=rw- lustre-MDT0001.19 10OPEN 13:02:01.675837453 2025.06.02 0x43 t=[0x240000bd3:0x1f:0x0] j=bash.0 ef=0x17 u=0:0 nid=192.168.202.26@tcp m=rw- lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0001.20 10OPEN 13:02:01.997569038 2025.06.02 0x1 t=[0x240000bd3:0x1f:0x0] j=cat.0 ef=0x17 u=0:0 nid=192.168.202.26@tcp m=r-- p=[0x240000bd3:0x1e:0x0] lustre-MDT0001.21 11CLOSE 13:02:02.004051043 2025.06.02 0x1 t=[0x240000bd3:0x1f:0x0] j=cat.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp lustre-MDT0001.20 10OPEN 13:02:01.997569038 2025.06.02 0x1 t=[0x240000bd3:0x1f:0x0] j=cat.0 ef=0x17 u=0:0 nid=192.168.202.26@tcp m=r-- p=[0x240000bd3:0x1e:0x0] lustre-MDT0001.21 11CLOSE 13:02:02.004051043 2025.06.02 0x1 t=[0x240000bd3:0x1f:0x0] j=cat.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0001: clear the changelog for cl6 of all records running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cat] [/mnt/lustre/d605.sanity-hsm/f605.sanity-hsm] lustre-MDT0001.22 10OPEN 13:02:02.430173900 2025.06.02 0x1 t=[0x240000bd3:0x1f:0x0] j=cat.500 ef=0x17 u=500:500 nid=192.168.202.26@tcp m=r-- lustre-MDT0001.23 11CLOSE 13:02:02.433457592 2025.06.02 0x1 t=[0x240000bd3:0x1f:0x0] j=cat.500 ef=0x13 u=500:500 nid=192.168.202.26@tcp lustre-MDT0001.22 10OPEN 13:02:02.430173900 2025.06.02 0x1 t=[0x240000bd3:0x1f:0x0] j=cat.500 ef=0x17 u=500:500 nid=192.168.202.26@tcp m=r-- lustre-MDT0001.23 11CLOSE 13:02:02.433457592 2025.06.02 0x1 t=[0x240000bd3:0x1f:0x0] j=cat.500 ef=0x13 u=500:500 nid=192.168.202.26@tcp lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0001.24 11CLOSE 13:02:02.622638905 2025.06.02 0x43 t=[0x240000bd3:0x1f:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp lustre-MDT0001.24 11CLOSE 13:02:02.622638905 2025.06.02 0x43 t=[0x240000bd3:0x1f:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.202.26@tcp lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0001: Deregistered changelog user #6 lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0000: Deregistered changelog user #6 PASS 605 (6s) == sanity-hsm test 606: llog_reader groks changelog fields ========================================================== 09:02:07 (1748869327) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl7 cl7' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d606.sanity-hsm/f606.sanity-hsm] Stopping /mnt/lustre-mds1 (opts:) on oleg226-server rec #2 type=1064553b len=64 offset 8256 Header size : 8192 llh_size : 64 Time : Mon Jun 2 08:49:30 2025 Number of records: 1 cat_idx: 1 last_idx: 2 Flags: 0x2012 Target uuid : ----------------------- #02 (064)fid=[0x1:0x3b:0x0] path=O/1/d27/59 Got UID/GID 500:500 Got NID 192.168.202.26@tcp Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0000 lustre-MDT0001: clear the changelog for cl7 of all records lustre-MDT0001: Deregistered changelog user #7 lustre-MDT0000: clear the changelog for cl7 of all records lustre-MDT0000: Deregistered changelog user #7 PASS 606 (15s) == sanity-hsm test 607a: release a file that was migrated after being archived ========================================================== 09:02:22 (1748869342) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0584228 s, 17.9 MB/s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_607a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' PASS 607a (5s) == sanity-hsm test 607b: Migrate should not change the HSM attribute of dirty files ========================================================== 09:02:28 (1748869348) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0589174 s, 17.8 MB/s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_607b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' llite.lustre-ffff8800a514a000.xattr_cache=0 llite.lustre-ffff8800b6d11000.xattr_cache=0 /mnt/lustre/d607b.sanity-hsm/f607b.sanity-hsm: (0x0000000b) exists dirty archived, archive_id:1 llite.lustre-ffff8800a514a000.xattr_cache=1 llite.lustre-ffff8800b6d11000.xattr_cache=1 PASS 607b (5s) == sanity-hsm test 607c: 'lfs swap_layouts' should set dirty flag on HSM file ========================================================== 09:02:33 (1748869353) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0561435 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0613039 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.056249 s, 18.6 MB/s Starting copytool 'agt1' on 'oleg226-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_607c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' lfs swap_layouts: error: cannot swap layout between '/mnt/lustre/d607c.sanity-hsm/f607c.sanity-hsm-1' and '/mnt/lustre/d607c.sanity-hsm/f607c.sanity-hsm-3': Operation not permitted (1) PASS 607c (5s) == sanity-hsm test complete, duration 2055 sec =========== 09:02:38 (1748869358) === sanity-hsm: start cleanup 09:02:39 (1748869359) === Stopping clients: oleg226-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg226-client.virtnet /mnt/lustre2 opts: === sanity-hsm: finish cleanup 09:02:46 (1748869366) ===