-----============= acceptance-small: sanity-hsm ============----- Mon Mar 16 14:23:51 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg434-server: /home/green/git/lustre-release/lustre/tests/except/sanity-hsm.ex - see MDS1_VERSION <= CLIENT_VERSION (34681496 <= 34681496) for LU-18924, go 100 105 excepting tests: === sanity-hsm: start setup 14:24:05 (1773685445) === Starting client oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre2 Mount client oleg434-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre2 Started clients oleg434-client.virtnet: 192.168.204.134@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg434-client.virtnet: executing check_config_client /mnt/lustre oleg434-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg434-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff917ac9c5d800.idle_timeout=debug osc.lustre-OST0000-osc-ffff917ad87b8000.idle_timeout=debug osc.lustre-OST0001-osc-ffff917ac9c5d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff917ad87b8000.idle_timeout=debug disable quota as required oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-hsm: finish setup 14:24:29 (1773685469) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8346] nobody:x:65534: Killing existing copytools on oleg434-client.virtnet pkill: pidfile not valid Try `pkill --help' for more information. Set HSM on and start Set sanity-hsm HSM policy == sanity-hsm test 1A: lfs hsm flags root/non-root access ========================================================== 14:24:55 (1773685495) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_set] [--norelease] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_clear] [--norelease] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_set] [--exists] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] lfs hsm_set: change hsm flags for '/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm' failed: Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] PASS 1A (4s) == sanity-hsm test 1a: mmap & cat a HSM released file ==== 14:24:59 (1773685499) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133268 s, 7.9 MB/s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_1a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Verifying released state: PASS 1a (8s) == sanity-hsm test 1b: Archive, Release and Restore composite file ========================================================== 14:25:07 (1773685507) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290495 s, 36.1 MB/s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_1b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' archive /mnt/lustre/d1b.sanity-hsm/f1b.sanity-hsm release /mnt/lustre/d1b.sanity-hsm/f1b.sanity-hsm verify released state: pass restore /mnt/lustre/d1b.sanity-hsm/f1b.sanity-hsm Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' verify restored state: pass PASS 1b (6s) == sanity-hsm test 1c: Check setting archive-id in lfs hsm_set ========================================================== 14:25:13 (1773685513) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d1c.sanity-hsm/f1c.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1c.sanity-hsm/f1c.sanity-hsm] verifying archive number is 32 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1c.sanity-hsm/f1c.sanity-hsm] verifying archive number is still 32 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1c.sanity-hsm/f1c.sanity-hsm] verifying archive number is 33 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1c.sanity-hsm/f1c.sanity-hsm] verifying archive number is 16 PASS 1c (4s) == sanity-hsm test 1d: Archive, Release and Restore DoM file ========================================================== 14:25:17 (1773685517) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323815 s, 32.4 MB/s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_1d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' archive /mnt/lustre/d1d.sanity-hsm/f1d.sanity-hsm Waiting 200s for 'SUCCEED' release /mnt/lustre/d1d.sanity-hsm/f1d.sanity-hsm verify released state: pass restore /mnt/lustre/d1d.sanity-hsm/f1d.sanity-hsm verify restored state: pass PASS 1d (6s) == sanity-hsm test 1e: Archive, Release and Restore SEL file ========================================================== 14:25:23 (1773685523) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334806 s, 31.3 MB/s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_1e/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' archive /mnt/lustre/d1e.sanity-hsm/f1e.sanity-hsm release /mnt/lustre/d1e.sanity-hsm/f1e.sanity-hsm verify released state: pass restore /mnt/lustre/d1e.sanity-hsm/f1e.sanity-hsm Waiting 200s for 'SUCCEED' verify restored state: pass PASS 1e (7s) == sanity-hsm test 1f: DoM file release after restore ==== 14:25:30 (1773685530) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376544 s, 27.8 MB/s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_1f/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' archive /mnt/lustre/d1f.sanity-hsm/f1f.sanity-hsm Waiting 200s for 'SUCCEED' release /mnt/lustre/d1f.sanity-hsm/f1f.sanity-hsm verify released state: pass restore /mnt/lustre/d1f.sanity-hsm/f1f.sanity-hsm Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' verify restored state: pass release again /mnt/lustre/d1f.sanity-hsm/f1f.sanity-hsm /mnt/lustre/d1f.sanity-hsm/f1f.sanity-hsm: (0x0000000d) released exists archived, archive_id:1 verify released state: pass PASS 1f (8s) == sanity-hsm test 2: Check file dirtyness when doing setattr ========================================================== 14:25:38 (1773685538) 0+0 records in 0+0 records out 0 bytes copied, 0.00509182 s, 0.0 kB/s 0x200000403:0x8:0x0 PASS 2 (4s) == sanity-hsm test 3: Check file dirtyness when opening for write ========================================================== 14:25:42 (1773685542) Succeed in opening file "/mnt/lustre/d3.sanity-hsm/f3.sanity-hsm"(flags=O_WRONLY) 3+0 records in 3+0 records out 3+0 records in 3+0 records out PASS 3 (3s) == sanity-hsm test 4: Useless cancel must not be registered ========================================================== 14:25:45 (1773685545) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814184 s, 12.9 MB/s PASS 4 (4s) == sanity-hsm test 8: Test default archive number ======== 14:25:49 (1773685549) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_8/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' PASS 8 (5s) == sanity-hsm test 9A: Use of explicit archive number, with dedicated copytool ========================================================== 14:25:54 (1773685554) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_9A/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 3 "/mnt/lustre2"' oleg434-client.virtnet: executing pgrep --pidfile=/var/run/lhsmtool_posix.pid --list-full hsmtool oleg434-client.virtnet: executing get_client_uuid /mnt/lustre2 found agent uuid=2242dc0c-b366-4b7b-bf7a-e0464872b03c archive_id=3 requests=[current:0 ok:0 errors:0] on mds1 found agent uuid=2242dc0c-b366-4b7b-bf7a-e0464872b03c archive_id=3 requests=[current:0 ok:0 errors:0] on mds2 PASS 9A (13s) == sanity-hsm test 9a: Multiple remote agents ============ 14:26:07 (1773685567) SKIP: sanity-hsm test_9a Need 3 or more clients, have 1 SKIP 9a (2s) == sanity-hsm test 10a: Archive a file =================== 14:26:09 (1773685569) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_10a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh: line 938: warning: command substitution: ignored null byte in input Verifying content Verifying hsm state Verifying archive number is 2 PASS 10a (7s) == sanity-hsm test 10b: Archive of non dirty file must work without doing request ========================================================== 14:26:16 (1773685576) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_10b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' PASS 10b (6s) == sanity-hsm test 10c: Check forbidden archive ========== 14:26:22 (1773685582) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_10c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Cannot send HSM request (use of /mnt/lustre/d10c.sanity-hsm/f10c.sanity-hsm): Operation not permitted PASS 10c (4s) == sanity-hsm test 10d: Archive a file on the default archive id ========================================================== 14:26:26 (1773685586) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_10d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' PASS 10d (6s) == sanity-hsm test 11a: Import a file ==================== 14:26:32 (1773685592) lhsmtool_posix: 1773685594.189170 [22145]: action=1 src=d11a.sanity-hsm/f11a.sanity-hsm dst=/mnt/lustre/d11a.sanity-hsm/f11a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1773685594.195319 [22145]: importing '/mnt/lustre/d11a.sanity-hsm/f11a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_11a//d11a.sanity-hsm/f11a.sanity-hsm' lhsmtool_posix: 1773685594.241295 [22145]: imported '/mnt/lustre/d11a.sanity-hsm/f11a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_11a//0424/0x200000402:0x26:0x0'=='/tmp/arc1/sanity-hsm.test_11a//d11a.sanity-hsm/f11a.sanity-hsm' lhsmtool_posix: 1773685594.241373 [22145]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Verifying released state: Verifying imported size 158=158 Verifying released pattern: raid0,released Verifying new fid 0x200000402:0x26:0x0 in archive PASS 11a (5s) == sanity-hsm test 11b: Import a deleted file using its FID ========================================================== 14:26:37 (1773685597) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_11b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' lhsmtool_posix: 1773685600.968237 [22845]: action=1 src=0x200000403:0x11:0x0 dst=/mnt/lustre/d11b.sanity-hsm/f11b.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1773685600.977813 [22845]: Resolving [0x200000403:0x11:0x0] to /tmp/arc1/sanity-hsm.test_11b//0412/0x200000403:0x11:0x0 lhsmtool_posix: 1773685600.978343 [22845]: importing '/mnt/lustre/d11b.sanity-hsm/f11b.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_11b//0412/0x200000403:0x11:0x0' lhsmtool_posix: 1773685601.004252 [22845]: imported '/mnt/lustre/d11b.sanity-hsm/f11b.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_11b//042a/0x200000402:0x28:0x0'=='/tmp/arc1/sanity-hsm.test_11b//0412/0x200000403:0x11:0x0' lhsmtool_posix: 1773685601.004294 [22845]: process finished, errs: 0 major, 0 minor, rc=0 (Success) /mnt/lustre/d11b.sanity-hsm/f11b.sanity-hsm: OK PASS 11b (7s) == sanity-hsm test 11c: Import a file to a directory with a pool ========================================================== 14:26:44 (1773685604) Creating new pool test_11c oleg434-server: Pool lustre.test_11c created Adding targets to pool oleg434-server: OST lustre-OST0001_UUID added to pool lustre.test_11c lhsmtool_posix: 1773685613.180826 [23560]: action=1 src=d11c.sanity-hsm/f11c.sanity-hsm dst=/mnt/lustre/d11c.sanity-hsm/f11c.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1773685613.191315 [23560]: importing '/mnt/lustre/d11c.sanity-hsm/f11c.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_11c//d11c.sanity-hsm/f11c.sanity-hsm' lhsmtool_posix: 1773685613.259838 [23560]: imported '/mnt/lustre/d11c.sanity-hsm/f11c.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_11c//0404/0x240000402:0x6:0x0'=='/tmp/arc1/sanity-hsm.test_11c//d11c.sanity-hsm/f11c.sanity-hsm' lhsmtool_posix: 1773685613.259908 [23560]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Destroy the created pools: test_11c lustre.test_11c oleg434-server: OST lustre-OST0001_UUID removed from pool lustre.test_11c oleg434-server: Pool lustre.test_11c destroyed PASS 11c (18s) == sanity-hsm test 12a: Restore an imported file explicitly ========================================================== 14:27:02 (1773685622) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1773685623.644611 [24296]: action=1 src=d12a.sanity-hsm/f12a.sanity-hsm dst=/mnt/lustre/d12a.sanity-hsm/f12a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1773685623.650556 [24296]: importing '/mnt/lustre/d12a.sanity-hsm/f12a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12a//d12a.sanity-hsm/f12a.sanity-hsm' lhsmtool_posix: 1773685623.703901 [24296]: imported '/mnt/lustre/d12a.sanity-hsm/f12a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12a//0429/0x200000402:0x2b:0x0'=='/tmp/arc1/sanity-hsm.test_12a//d12a.sanity-hsm/f12a.sanity-hsm' lhsmtool_posix: 1773685623.703955 [24296]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Verifying released state: Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Verifying file state: PASS 12a (6s) == sanity-hsm test 12b: Restore an imported file implicitly ========================================================== 14:27:08 (1773685628) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1773685630.221707 [25010]: action=1 src=d12b.sanity-hsm/f12b.sanity-hsm dst=/mnt/lustre/d12b.sanity-hsm/f12b.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1773685630.233159 [25010]: importing '/mnt/lustre/d12b.sanity-hsm/f12b.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12b//d12b.sanity-hsm/f12b.sanity-hsm' lhsmtool_posix: 1773685630.286632 [25010]: imported '/mnt/lustre/d12b.sanity-hsm/f12b.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12b//042c/0x200000402:0x2e:0x0'=='/tmp/arc1/sanity-hsm.test_12b//d12b.sanity-hsm/f12b.sanity-hsm' lhsmtool_posix: 1773685630.289180 [25010]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Verifying released state: Verifying file state after restore: PASS 12b (6s) == sanity-hsm test 12c: Restore a file with stripe of 2 == 14:27:14 (1773685634) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.0712992 s, 73.5 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' /mnt/lustre/d12c.sanity-hsm/f12c.sanity-hsm: OK PASS 12c (7s) == sanity-hsm test 12d: Restore of a non archived, non released file must work ========================================================== 14:27:21 (1773685641) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' PASS 12d (7s) == sanity-hsm test 12e: Check forbidden restore ========== 14:27:28 (1773685648) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12e/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' /mnt/lustre/d12e.sanity-hsm/f12e.sanity-hsm: (0x0000000b) exists dirty archived, archive_id:1 Cannot send HSM request (use of /mnt/lustre/d12e.sanity-hsm/f12e.sanity-hsm): Operation not permitted PASS 12e (5s) == sanity-hsm test 12f: Restore a released file explicitly ========================================================== 14:27:33 (1773685653) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12f/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Waiting 200s for 'SUCCEED' Verifying file state: PASS 12f (9s) == sanity-hsm test 12g: Restore a released file implicitly ========================================================== 14:27:42 (1773685662) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12g/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' PASS 12g (7s) == sanity-hsm test 12h: Restore a released file implicitly from a second node ========================================================== 14:27:49 (1773685669) SKIP: sanity-hsm test_12h Need 2 or more clients, have 1 SKIP 12h (2s) == sanity-hsm test 12m: Archive/release/implicit restore ========================================================== 14:27:51 (1773685671) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12m/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' PASS 12m (7s) == sanity-hsm test 12n: Import/implicit restore/release == 14:27:58 (1773685678) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12n/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1773685679.623487 [29870]: action=1 src=d12n.sanity-hsm/f12n.sanity-hsm dst=/mnt/lustre/d12n.sanity-hsm/f12n.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1773685679.626719 [29870]: importing '/mnt/lustre/d12n.sanity-hsm/f12n.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12n//d12n.sanity-hsm/f12n.sanity-hsm' lhsmtool_posix: 1773685679.657613 [29870]: imported '/mnt/lustre/d12n.sanity-hsm/f12n.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12n//043e/0x200000402:0x3c:0x0'=='/tmp/arc1/sanity-hsm.test_12n//d12n.sanity-hsm/f12n.sanity-hsm' lhsmtool_posix: 1773685679.657956 [29870]: process finished, errs: 0 major, 0 minor, rc=0 (Success) PASS 12n (5s) == sanity-hsm test 12o: Layout-swap failure during Restore leaves file released ========================================================== 14:28:03 (1773685683) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12o/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' fail_loc=0x152 mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0001.hsm.policy=+NRA diff: /mnt/lustre/d12o.sanity-hsm/f12o.sanity-hsm: No data available mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA fail_loc=0 mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge PASS 12o (25s) == sanity-hsm test 12p: implicit restore of a file on copytool mount point ========================================================== 14:28:28 (1773685708) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12p/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' PASS 12p (7s) == sanity-hsm test 12q: file attributes are refreshed after restore ========================================================== 14:28:35 (1773685715) Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre3 Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12q/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre3"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17164 s, 6.1 MB/s Killing existing copytools on oleg434-client.virtnet copytools stopped in 0s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12q/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre3"' Stopping client oleg434-client.virtnet /mnt/lustre3 (opts:) PASS 12q (12s) == sanity-hsm test 12r: lseek restores released file ===== 14:28:47 (1773685727) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12r/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' PASS 12r (5s) == sanity-hsm test 12s: race between restore requests ==== 14:28:52 (1773685732) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12s/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' fail_loc=0x8000018b PASS 12s (11s) == sanity-hsm test 12t: Multiple parallel reads for a HSM imported file ========================================================== 14:29:03 (1773685743) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12t/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1773685744.775928 [34009]: action=1 src=0x200000403:0x2b:0x0 dst=/mnt/lustre/d12t.sanity-hsm/f12t.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1773685744.783477 [34009]: Resolving [0x200000403:0x2b:0x0] to /tmp/arc1/sanity-hsm.test_12t//0428/0x200000403:0x2b:0x0 lhsmtool_posix: 1773685744.783575 [34009]: importing '/mnt/lustre/d12t.sanity-hsm/f12t.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12t//0428/0x200000403:0x2b:0x0' lhsmtool_posix: 1773685744.822463 [34009]: imported '/mnt/lustre/d12t.sanity-hsm/f12t.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12t//0448/0x200000402:0x4a:0x0'=='/tmp/arc1/sanity-hsm.test_12t//0428/0x200000403:0x2b:0x0' lhsmtool_posix: 1773685744.823560 [34009]: process finished, errs: 0 major, 0 minor, rc=0 (Success) PASS 12t (5s) == sanity-hsm test 12u: Multiple reads on multiple HSM imported files in parallel ========================================================== 14:29:08 (1773685748) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12u/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1773685756.002754 [34881]: action=1 src=0x200000403:0x2d:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.0 mount_point=/mnt/lustre2 lhsmtool_posix: 1773685756.008177 [34881]: Resolving [0x200000403:0x2d:0x0] to /tmp/arc1/sanity-hsm.test_12u//042e/0x200000403:0x2d:0x0 lhsmtool_posix: 1773685756.008240 [34881]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.0' from '/tmp/arc1/sanity-hsm.test_12u//042e/0x200000403:0x2d:0x0' lhsmtool_posix: 1773685756.034385 [34881]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.0' from '/tmp/arc1/sanity-hsm.test_12u//044e/0x200000402:0x4c:0x0'=='/tmp/arc1/sanity-hsm.test_12u//042e/0x200000403:0x2d:0x0' lhsmtool_posix: 1773685756.034441 [34881]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1773685756.173194 [34910]: action=1 src=0x200000403:0x2e:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.1 mount_point=/mnt/lustre2 lhsmtool_posix: 1773685756.176790 [34910]: Resolving [0x200000403:0x2e:0x0] to /tmp/arc1/sanity-hsm.test_12u//042d/0x200000403:0x2e:0x0 lhsmtool_posix: 1773685756.176864 [34910]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.1' from '/tmp/arc1/sanity-hsm.test_12u//042d/0x200000403:0x2e:0x0' lhsmtool_posix: 1773685756.203624 [34910]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.1' from '/tmp/arc1/sanity-hsm.test_12u//044f/0x200000402:0x4d:0x0'=='/tmp/arc1/sanity-hsm.test_12u//042d/0x200000403:0x2e:0x0' lhsmtool_posix: 1773685756.203680 [34910]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1773685756.388311 [34939]: action=1 src=0x200000403:0x2f:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.2 mount_point=/mnt/lustre2 lhsmtool_posix: 1773685756.393054 [34939]: Resolving [0x200000403:0x2f:0x0] to /tmp/arc1/sanity-hsm.test_12u//042c/0x200000403:0x2f:0x0 lhsmtool_posix: 1773685756.393132 [34939]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.2' from '/tmp/arc1/sanity-hsm.test_12u//042c/0x200000403:0x2f:0x0' lhsmtool_posix: 1773685756.414179 [34939]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.2' from '/tmp/arc1/sanity-hsm.test_12u//044c/0x200000402:0x4e:0x0'=='/tmp/arc1/sanity-hsm.test_12u//042c/0x200000403:0x2f:0x0' lhsmtool_posix: 1773685756.414221 [34939]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1773685756.544090 [34968]: action=1 src=0x200000403:0x30:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.3 mount_point=/mnt/lustre2 lhsmtool_posix: 1773685756.548464 [34968]: Resolving [0x200000403:0x30:0x0] to /tmp/arc1/sanity-hsm.test_12u//0433/0x200000403:0x30:0x0 lhsmtool_posix: 1773685756.548543 [34968]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.3' from '/tmp/arc1/sanity-hsm.test_12u//0433/0x200000403:0x30:0x0' lhsmtool_posix: 1773685756.563958 [34968]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.3' from '/tmp/arc1/sanity-hsm.test_12u//044d/0x200000402:0x4f:0x0'=='/tmp/arc1/sanity-hsm.test_12u//0433/0x200000403:0x30:0x0' lhsmtool_posix: 1773685756.564010 [34968]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1773685756.694883 [34997]: action=1 src=0x200000403:0x31:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.4 mount_point=/mnt/lustre2 lhsmtool_posix: 1773685756.701800 [34997]: Resolving [0x200000403:0x31:0x0] to /tmp/arc1/sanity-hsm.test_12u//0432/0x200000403:0x31:0x0 lhsmtool_posix: 1773685756.701890 [34997]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.4' from '/tmp/arc1/sanity-hsm.test_12u//0432/0x200000403:0x31:0x0' lhsmtool_posix: 1773685756.730646 [34997]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.4' from '/tmp/arc1/sanity-hsm.test_12u//0452/0x200000402:0x50:0x0'=='/tmp/arc1/sanity-hsm.test_12u//0432/0x200000403:0x31:0x0' lhsmtool_posix: 1773685756.730706 [34997]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1773685756.877468 [35026]: action=1 src=0x200000403:0x32:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.5 mount_point=/mnt/lustre2 lhsmtool_posix: 1773685756.882292 [35026]: Resolving [0x200000403:0x32:0x0] to /tmp/arc1/sanity-hsm.test_12u//0431/0x200000403:0x32:0x0 lhsmtool_posix: 1773685756.882373 [35026]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.5' from '/tmp/arc1/sanity-hsm.test_12u//0431/0x200000403:0x32:0x0' lhsmtool_posix: 1773685756.912287 [35026]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.5' from '/tmp/arc1/sanity-hsm.test_12u//0453/0x200000402:0x51:0x0'=='/tmp/arc1/sanity-hsm.test_12u//0431/0x200000403:0x32:0x0' lhsmtool_posix: 1773685756.912335 [35026]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1773685757.051900 [35055]: action=1 src=0x200000403:0x33:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.6 mount_point=/mnt/lustre2 lhsmtool_posix: 1773685757.055499 [35055]: Resolving [0x200000403:0x33:0x0] to /tmp/arc1/sanity-hsm.test_12u//0430/0x200000403:0x33:0x0 lhsmtool_posix: 1773685757.055648 [35055]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.6' from '/tmp/arc1/sanity-hsm.test_12u//0430/0x200000403:0x33:0x0' lhsmtool_posix: 1773685757.083602 [35055]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.6' from '/tmp/arc1/sanity-hsm.test_12u//0450/0x200000402:0x52:0x0'=='/tmp/arc1/sanity-hsm.test_12u//0430/0x200000403:0x33:0x0' lhsmtool_posix: 1773685757.083662 [35055]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1773685757.218538 [35084]: action=1 src=0x200000403:0x34:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.7 mount_point=/mnt/lustre2 lhsmtool_posix: 1773685757.223867 [35084]: Resolving [0x200000403:0x34:0x0] to /tmp/arc1/sanity-hsm.test_12u//0437/0x200000403:0x34:0x0 lhsmtool_posix: 1773685757.223943 [35084]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.7' from '/tmp/arc1/sanity-hsm.test_12u//0437/0x200000403:0x34:0x0' lhsmtool_posix: 1773685757.253648 [35084]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.7' from '/tmp/arc1/sanity-hsm.test_12u//0451/0x200000402:0x53:0x0'=='/tmp/arc1/sanity-hsm.test_12u//0437/0x200000403:0x34:0x0' lhsmtool_posix: 1773685757.253696 [35084]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1773685757.403752 [35113]: action=1 src=0x200000403:0x35:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.8 mount_point=/mnt/lustre2 lhsmtool_posix: 1773685757.407997 [35113]: Resolving [0x200000403:0x35:0x0] to /tmp/arc1/sanity-hsm.test_12u//0436/0x200000403:0x35:0x0 lhsmtool_posix: 1773685757.408119 [35113]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.8' from '/tmp/arc1/sanity-hsm.test_12u//0436/0x200000403:0x35:0x0' lhsmtool_posix: 1773685757.439482 [35113]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.8' from '/tmp/arc1/sanity-hsm.test_12u//0456/0x200000402:0x54:0x0'=='/tmp/arc1/sanity-hsm.test_12u//0436/0x200000403:0x35:0x0' lhsmtool_posix: 1773685757.439535 [35113]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1773685757.613460 [35142]: action=1 src=0x200000403:0x36:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.9 mount_point=/mnt/lustre2 lhsmtool_posix: 1773685757.617610 [35142]: Resolving [0x200000403:0x36:0x0] to /tmp/arc1/sanity-hsm.test_12u//0435/0x200000403:0x36:0x0 lhsmtool_posix: 1773685757.618010 [35142]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.9' from '/tmp/arc1/sanity-hsm.test_12u//0435/0x200000403:0x36:0x0' lhsmtool_posix: 1773685757.649305 [35142]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.9' from '/tmp/arc1/sanity-hsm.test_12u//0457/0x200000402:0x55:0x0'=='/tmp/arc1/sanity-hsm.test_12u//0435/0x200000403:0x36:0x0' lhsmtool_posix: 1773685757.649376 [35142]: process finished, errs: 0 major, 0 minor, rc=0 (Success) PASS 12u (14s) == sanity-hsm test 13: Recursively import and restore a directory ========================================================== 14:29:22 (1773685762) lhsmtool_posix: 1773685767.167755 [36894]: action=1 src=subdir dst=/mnt/lustre/d13.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1773685767.313588 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.10' lhsmtool_posix: 1773685767.380303 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.10' from '/tmp/arc1/sanity-hsm.test_13//0455/0x200000402:0x57:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.10' lhsmtool_posix: 1773685767.380819 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.9' lhsmtool_posix: 1773685767.418886 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.9' from '/tmp/arc1/sanity-hsm.test_13//045a/0x200000402:0x58:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.9' lhsmtool_posix: 1773685767.419141 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.8' lhsmtool_posix: 1773685767.452358 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.8' from '/tmp/arc1/sanity-hsm.test_13//045b/0x200000402:0x59:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.8' lhsmtool_posix: 1773685767.453051 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.7' lhsmtool_posix: 1773685767.478998 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.7' from '/tmp/arc1/sanity-hsm.test_13//0458/0x200000402:0x5a:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.7' lhsmtool_posix: 1773685767.479299 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.6' lhsmtool_posix: 1773685767.507542 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.6' from '/tmp/arc1/sanity-hsm.test_13//0459/0x200000402:0x5b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.6' lhsmtool_posix: 1773685767.507799 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.5' lhsmtool_posix: 1773685767.533964 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.5' from '/tmp/arc1/sanity-hsm.test_13//045e/0x200000402:0x5c:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.5' lhsmtool_posix: 1773685767.534241 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.4' lhsmtool_posix: 1773685767.556143 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.4' from '/tmp/arc1/sanity-hsm.test_13//045f/0x200000402:0x5d:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.4' lhsmtool_posix: 1773685767.556372 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.3' lhsmtool_posix: 1773685767.582834 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.3' from '/tmp/arc1/sanity-hsm.test_13//045c/0x200000402:0x5e:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.3' lhsmtool_posix: 1773685767.583406 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.2' lhsmtool_posix: 1773685767.614680 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.2' from '/tmp/arc1/sanity-hsm.test_13//045d/0x200000402:0x5f:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.2' lhsmtool_posix: 1773685767.614873 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.1' lhsmtool_posix: 1773685767.644188 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.1' from '/tmp/arc1/sanity-hsm.test_13//0462/0x200000402:0x60:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.1' lhsmtool_posix: 1773685767.648824 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.10' lhsmtool_posix: 1773685767.682284 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.10' from '/tmp/arc1/sanity-hsm.test_13//040e/0x240000402:0xc:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.10' lhsmtool_posix: 1773685767.682555 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.9' lhsmtool_posix: 1773685767.727697 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.9' from '/tmp/arc1/sanity-hsm.test_13//040f/0x240000402:0xd:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.9' lhsmtool_posix: 1773685767.727947 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.8' lhsmtool_posix: 1773685767.758412 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.8' from '/tmp/arc1/sanity-hsm.test_13//040c/0x240000402:0xe:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.8' lhsmtool_posix: 1773685767.758618 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.7' lhsmtool_posix: 1773685767.800214 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.7' from '/tmp/arc1/sanity-hsm.test_13//040d/0x240000402:0xf:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.7' lhsmtool_posix: 1773685767.800474 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.6' lhsmtool_posix: 1773685767.835097 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.6' from '/tmp/arc1/sanity-hsm.test_13//0412/0x240000402:0x10:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.6' lhsmtool_posix: 1773685767.835348 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.5' lhsmtool_posix: 1773685767.867769 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.5' from '/tmp/arc1/sanity-hsm.test_13//0413/0x240000402:0x11:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.5' lhsmtool_posix: 1773685767.868073 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.4' lhsmtool_posix: 1773685767.920129 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.4' from '/tmp/arc1/sanity-hsm.test_13//0410/0x240000402:0x12:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.4' lhsmtool_posix: 1773685767.920360 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.3' lhsmtool_posix: 1773685767.953835 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.3' from '/tmp/arc1/sanity-hsm.test_13//0411/0x240000402:0x13:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.3' lhsmtool_posix: 1773685767.955631 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.2' lhsmtool_posix: 1773685767.986318 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.2' from '/tmp/arc1/sanity-hsm.test_13//0416/0x240000402:0x14:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.2' lhsmtool_posix: 1773685767.986581 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.1' lhsmtool_posix: 1773685768.010603 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.1' from '/tmp/arc1/sanity-hsm.test_13//0417/0x240000402:0x15:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.1' lhsmtool_posix: 1773685768.018206 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.10' lhsmtool_posix: 1773685768.066254 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.10' from '/tmp/arc1/sanity-hsm.test_13//0460/0x200000402:0x62:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.10' lhsmtool_posix: 1773685768.067059 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.9' lhsmtool_posix: 1773685768.091583 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.9' from '/tmp/arc1/sanity-hsm.test_13//0461/0x200000402:0x63:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.9' lhsmtool_posix: 1773685768.091956 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.8' lhsmtool_posix: 1773685768.110389 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.8' from '/tmp/arc1/sanity-hsm.test_13//0466/0x200000402:0x64:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.8' lhsmtool_posix: 1773685768.110617 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.7' lhsmtool_posix: 1773685768.134358 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.7' from '/tmp/arc1/sanity-hsm.test_13//0467/0x200000402:0x65:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.7' lhsmtool_posix: 1773685768.134576 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.6' lhsmtool_posix: 1773685768.160135 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.6' from '/tmp/arc1/sanity-hsm.test_13//0464/0x200000402:0x66:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.6' lhsmtool_posix: 1773685768.160363 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.5' lhsmtool_posix: 1773685768.188762 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.5' from '/tmp/arc1/sanity-hsm.test_13//0465/0x200000402:0x67:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.5' lhsmtool_posix: 1773685768.189707 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.4' lhsmtool_posix: 1773685768.218737 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.4' from '/tmp/arc1/sanity-hsm.test_13//046a/0x200000402:0x68:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.4' lhsmtool_posix: 1773685768.218977 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.3' lhsmtool_posix: 1773685768.242146 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.3' from '/tmp/arc1/sanity-hsm.test_13//046b/0x200000402:0x69:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.3' lhsmtool_posix: 1773685768.242341 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.2' lhsmtool_posix: 1773685768.259096 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.2' from '/tmp/arc1/sanity-hsm.test_13//0468/0x200000402:0x6a:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.2' lhsmtool_posix: 1773685768.259359 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.1' lhsmtool_posix: 1773685768.281700 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.1' from '/tmp/arc1/sanity-hsm.test_13//0469/0x200000402:0x6b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.1' lhsmtool_posix: 1773685768.287217 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.10' lhsmtool_posix: 1773685768.334171 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.10' from '/tmp/arc1/sanity-hsm.test_13//0415/0x240000402:0x17:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.10' lhsmtool_posix: 1773685768.339735 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.9' lhsmtool_posix: 1773685768.371287 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.9' from '/tmp/arc1/sanity-hsm.test_13//041a/0x240000402:0x18:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.9' lhsmtool_posix: 1773685768.371515 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.8' lhsmtool_posix: 1773685768.394196 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.8' from '/tmp/arc1/sanity-hsm.test_13//041b/0x240000402:0x19:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.8' lhsmtool_posix: 1773685768.394438 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.7' lhsmtool_posix: 1773685768.419136 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.7' from '/tmp/arc1/sanity-hsm.test_13//0418/0x240000402:0x1a:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.7' lhsmtool_posix: 1773685768.419339 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.6' lhsmtool_posix: 1773685768.452987 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.6' from '/tmp/arc1/sanity-hsm.test_13//0419/0x240000402:0x1b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.6' lhsmtool_posix: 1773685768.453267 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.5' lhsmtool_posix: 1773685768.490850 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.5' from '/tmp/arc1/sanity-hsm.test_13//041e/0x240000402:0x1c:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.5' lhsmtool_posix: 1773685768.491303 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.4' lhsmtool_posix: 1773685768.518911 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.4' from '/tmp/arc1/sanity-hsm.test_13//041f/0x240000402:0x1d:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.4' lhsmtool_posix: 1773685768.519492 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.3' lhsmtool_posix: 1773685768.541891 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.3' from '/tmp/arc1/sanity-hsm.test_13//041c/0x240000402:0x1e:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.3' lhsmtool_posix: 1773685768.542086 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.2' lhsmtool_posix: 1773685768.575395 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.2' from '/tmp/arc1/sanity-hsm.test_13//041d/0x240000402:0x1f:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.2' lhsmtool_posix: 1773685768.575623 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.1' lhsmtool_posix: 1773685768.604867 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.1' from '/tmp/arc1/sanity-hsm.test_13//0422/0x240000402:0x20:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.1' lhsmtool_posix: 1773685768.610293 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.10' lhsmtool_posix: 1773685768.644229 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.10' from '/tmp/arc1/sanity-hsm.test_13//0420/0x240000402:0x22:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.10' lhsmtool_posix: 1773685768.644445 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.9' lhsmtool_posix: 1773685768.675971 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.9' from '/tmp/arc1/sanity-hsm.test_13//0421/0x240000402:0x23:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.9' lhsmtool_posix: 1773685768.676406 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.8' lhsmtool_posix: 1773685768.709910 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.8' from '/tmp/arc1/sanity-hsm.test_13//0426/0x240000402:0x24:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.8' lhsmtool_posix: 1773685768.712207 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.7' lhsmtool_posix: 1773685768.738692 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.7' from '/tmp/arc1/sanity-hsm.test_13//0427/0x240000402:0x25:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.7' lhsmtool_posix: 1773685768.741250 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.6' lhsmtool_posix: 1773685768.779282 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.6' from '/tmp/arc1/sanity-hsm.test_13//0424/0x240000402:0x26:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.6' lhsmtool_posix: 1773685768.779779 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.5' lhsmtool_posix: 1773685768.810069 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.5' from '/tmp/arc1/sanity-hsm.test_13//0425/0x240000402:0x27:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.5' lhsmtool_posix: 1773685768.810233 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.4' lhsmtool_posix: 1773685768.832514 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.4' from '/tmp/arc1/sanity-hsm.test_13//042a/0x240000402:0x28:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.4' lhsmtool_posix: 1773685768.832735 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.3' lhsmtool_posix: 1773685768.852604 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.3' from '/tmp/arc1/sanity-hsm.test_13//042b/0x240000402:0x29:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.3' lhsmtool_posix: 1773685768.852788 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.2' lhsmtool_posix: 1773685768.881425 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.2' from '/tmp/arc1/sanity-hsm.test_13//0428/0x240000402:0x2a:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.2' lhsmtool_posix: 1773685768.881647 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.1' lhsmtool_posix: 1773685768.912343 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.1' from '/tmp/arc1/sanity-hsm.test_13//0429/0x240000402:0x2b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.1' lhsmtool_posix: 1773685768.915505 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.10' lhsmtool_posix: 1773685768.954153 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.10' from '/tmp/arc1/sanity-hsm.test_13//042f/0x240000402:0x2d:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.10' lhsmtool_posix: 1773685768.954339 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.9' lhsmtool_posix: 1773685768.986996 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.9' from '/tmp/arc1/sanity-hsm.test_13//042c/0x240000402:0x2e:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.9' lhsmtool_posix: 1773685768.987617 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.8' lhsmtool_posix: 1773685769.015148 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.8' from '/tmp/arc1/sanity-hsm.test_13//042d/0x240000402:0x2f:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.8' lhsmtool_posix: 1773685769.015493 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.7' lhsmtool_posix: 1773685769.041566 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.7' from '/tmp/arc1/sanity-hsm.test_13//0432/0x240000402:0x30:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.7' lhsmtool_posix: 1773685769.041828 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.6' lhsmtool_posix: 1773685769.077282 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.6' from '/tmp/arc1/sanity-hsm.test_13//0433/0x240000402:0x31:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.6' lhsmtool_posix: 1773685769.077511 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.5' lhsmtool_posix: 1773685769.108394 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.5' from '/tmp/arc1/sanity-hsm.test_13//0430/0x240000402:0x32:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.5' lhsmtool_posix: 1773685769.108611 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.4' lhsmtool_posix: 1773685769.151717 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.4' from '/tmp/arc1/sanity-hsm.test_13//0431/0x240000402:0x33:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.4' lhsmtool_posix: 1773685769.151910 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.3' lhsmtool_posix: 1773685769.177448 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.3' from '/tmp/arc1/sanity-hsm.test_13//0436/0x240000402:0x34:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.3' lhsmtool_posix: 1773685769.177635 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.2' lhsmtool_posix: 1773685769.201912 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.2' from '/tmp/arc1/sanity-hsm.test_13//0437/0x240000402:0x35:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.2' lhsmtool_posix: 1773685769.202131 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.1' lhsmtool_posix: 1773685769.220722 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.1' from '/tmp/arc1/sanity-hsm.test_13//0434/0x240000402:0x36:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.1' lhsmtool_posix: 1773685769.223850 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.10' lhsmtool_posix: 1773685769.256396 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.10' from '/tmp/arc1/sanity-hsm.test_13//043a/0x240000402:0x38:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.10' lhsmtool_posix: 1773685769.256552 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.9' lhsmtool_posix: 1773685769.288075 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.9' from '/tmp/arc1/sanity-hsm.test_13//043b/0x240000402:0x39:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.9' lhsmtool_posix: 1773685769.288240 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.8' lhsmtool_posix: 1773685769.313353 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.8' from '/tmp/arc1/sanity-hsm.test_13//0438/0x240000402:0x3a:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.8' lhsmtool_posix: 1773685769.313504 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.7' lhsmtool_posix: 1773685769.335006 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.7' from '/tmp/arc1/sanity-hsm.test_13//0439/0x240000402:0x3b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.7' lhsmtool_posix: 1773685769.335214 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.6' lhsmtool_posix: 1773685769.364414 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.6' from '/tmp/arc1/sanity-hsm.test_13//043e/0x240000402:0x3c:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.6' lhsmtool_posix: 1773685769.364633 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.5' lhsmtool_posix: 1773685769.400664 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.5' from '/tmp/arc1/sanity-hsm.test_13//043f/0x240000402:0x3d:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.5' lhsmtool_posix: 1773685769.401195 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.4' lhsmtool_posix: 1773685769.429938 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.4' from '/tmp/arc1/sanity-hsm.test_13//043c/0x240000402:0x3e:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.4' lhsmtool_posix: 1773685769.430757 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.3' lhsmtool_posix: 1773685769.470834 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.3' from '/tmp/arc1/sanity-hsm.test_13//043d/0x240000402:0x3f:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.3' lhsmtool_posix: 1773685769.471058 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.2' lhsmtool_posix: 1773685769.511982 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.2' from '/tmp/arc1/sanity-hsm.test_13//0442/0x240000402:0x40:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.2' lhsmtool_posix: 1773685769.512237 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.1' lhsmtool_posix: 1773685769.549303 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.1' from '/tmp/arc1/sanity-hsm.test_13//0443/0x240000402:0x41:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.1' lhsmtool_posix: 1773685769.560236 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.10' lhsmtool_posix: 1773685769.638954 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.10' from '/tmp/arc1/sanity-hsm.test_13//046f/0x200000402:0x6d:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.10' lhsmtool_posix: 1773685769.639198 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.9' lhsmtool_posix: 1773685769.668112 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.9' from '/tmp/arc1/sanity-hsm.test_13//046c/0x200000402:0x6e:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.9' lhsmtool_posix: 1773685769.668261 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.8' lhsmtool_posix: 1773685769.706474 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.8' from '/tmp/arc1/sanity-hsm.test_13//046d/0x200000402:0x6f:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.8' lhsmtool_posix: 1773685769.706985 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.7' lhsmtool_posix: 1773685769.745937 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.7' from '/tmp/arc1/sanity-hsm.test_13//0472/0x200000402:0x70:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.7' lhsmtool_posix: 1773685769.746293 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.6' lhsmtool_posix: 1773685769.779701 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.6' from '/tmp/arc1/sanity-hsm.test_13//0473/0x200000402:0x71:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.6' lhsmtool_posix: 1773685769.779892 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.5' lhsmtool_posix: 1773685769.803148 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.5' from '/tmp/arc1/sanity-hsm.test_13//0470/0x200000402:0x72:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.5' lhsmtool_posix: 1773685769.803369 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.4' lhsmtool_posix: 1773685769.836724 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.4' from '/tmp/arc1/sanity-hsm.test_13//0471/0x200000402:0x73:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.4' lhsmtool_posix: 1773685769.836931 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.3' lhsmtool_posix: 1773685769.863999 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.3' from '/tmp/arc1/sanity-hsm.test_13//0476/0x200000402:0x74:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.3' lhsmtool_posix: 1773685769.864220 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.2' lhsmtool_posix: 1773685769.901590 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.2' from '/tmp/arc1/sanity-hsm.test_13//0477/0x200000402:0x75:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.2' lhsmtool_posix: 1773685769.901776 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.1' lhsmtool_posix: 1773685769.930364 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.1' from '/tmp/arc1/sanity-hsm.test_13//0474/0x200000402:0x76:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.1' lhsmtool_posix: 1773685769.933995 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.10' lhsmtool_posix: 1773685769.961758 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.10' from '/tmp/arc1/sanity-hsm.test_13//0441/0x240000402:0x43:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.10' lhsmtool_posix: 1773685769.961954 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.9' lhsmtool_posix: 1773685769.990836 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.9' from '/tmp/arc1/sanity-hsm.test_13//0446/0x240000402:0x44:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.9' lhsmtool_posix: 1773685769.991446 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.8' lhsmtool_posix: 1773685770.015851 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.8' from '/tmp/arc1/sanity-hsm.test_13//0447/0x240000402:0x45:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.8' lhsmtool_posix: 1773685770.016096 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.7' lhsmtool_posix: 1773685770.040998 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.7' from '/tmp/arc1/sanity-hsm.test_13//0444/0x240000402:0x46:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.7' lhsmtool_posix: 1773685770.041228 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.6' lhsmtool_posix: 1773685770.058496 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.6' from '/tmp/arc1/sanity-hsm.test_13//0445/0x240000402:0x47:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.6' lhsmtool_posix: 1773685770.058671 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.5' lhsmtool_posix: 1773685770.077463 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.5' from '/tmp/arc1/sanity-hsm.test_13//044a/0x240000402:0x48:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.5' lhsmtool_posix: 1773685770.077662 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.4' lhsmtool_posix: 1773685770.091821 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.4' from '/tmp/arc1/sanity-hsm.test_13//044b/0x240000402:0x49:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.4' lhsmtool_posix: 1773685770.092035 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.3' lhsmtool_posix: 1773685770.109494 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.3' from '/tmp/arc1/sanity-hsm.test_13//0448/0x240000402:0x4a:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.3' lhsmtool_posix: 1773685770.109686 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.2' lhsmtool_posix: 1773685770.135150 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.2' from '/tmp/arc1/sanity-hsm.test_13//0449/0x240000402:0x4b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.2' lhsmtool_posix: 1773685770.135362 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.1' lhsmtool_posix: 1773685770.165296 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.1' from '/tmp/arc1/sanity-hsm.test_13//044e/0x240000402:0x4c:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.1' lhsmtool_posix: 1773685770.172858 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.10' lhsmtool_posix: 1773685770.238581 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.10' from '/tmp/arc1/sanity-hsm.test_13//047a/0x200000402:0x78:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.10' lhsmtool_posix: 1773685770.238793 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.9' lhsmtool_posix: 1773685770.256636 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.9' from '/tmp/arc1/sanity-hsm.test_13//047b/0x200000402:0x79:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.9' lhsmtool_posix: 1773685770.256822 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.8' lhsmtool_posix: 1773685770.277283 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.8' from '/tmp/arc1/sanity-hsm.test_13//0478/0x200000402:0x7a:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.8' lhsmtool_posix: 1773685770.277498 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.7' lhsmtool_posix: 1773685770.301077 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.7' from '/tmp/arc1/sanity-hsm.test_13//0479/0x200000402:0x7b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.7' lhsmtool_posix: 1773685770.301447 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.6' lhsmtool_posix: 1773685770.323870 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.6' from '/tmp/arc1/sanity-hsm.test_13//047e/0x200000402:0x7c:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.6' lhsmtool_posix: 1773685770.324078 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.5' lhsmtool_posix: 1773685770.351196 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.5' from '/tmp/arc1/sanity-hsm.test_13//047f/0x200000402:0x7d:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.5' lhsmtool_posix: 1773685770.351426 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.4' lhsmtool_posix: 1773685770.374767 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.4' from '/tmp/arc1/sanity-hsm.test_13//047c/0x200000402:0x7e:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.4' lhsmtool_posix: 1773685770.374961 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.3' lhsmtool_posix: 1773685770.395635 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.3' from '/tmp/arc1/sanity-hsm.test_13//047d/0x200000402:0x7f:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.3' lhsmtool_posix: 1773685770.395834 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.2' lhsmtool_posix: 1773685770.428715 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.2' from '/tmp/arc1/sanity-hsm.test_13//0482/0x200000402:0x80:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.2' lhsmtool_posix: 1773685770.429337 [36894]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.1' lhsmtool_posix: 1773685770.459870 [36894]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.1' from '/tmp/arc1/sanity-hsm.test_13//0483/0x200000402:0x81:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.1' lhsmtool_posix: 1773685770.459964 [36894]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_13/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' PASS 13 (17s) == sanity-hsm test 14: Rebind archived file to a new fid ========================================================== 14:29:39 (1773685779) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_14/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1252 s, 8.4 MB/s Waiting 200s for 'SUCCEED' 0+0 records in 0+0 records out 0 bytes copied, 0.0023601 s, 0.0 kB/s rebind 0x200000403:0x6a:0x0 to 0x200000403:0x6b:0x0 lhsmtool_posix: 1773685782.645357 [37851]: action=2 src=0x200000403:0x6a:0x0 dst=0x200000403:0x6b:0x0 mount_point=/mnt/lustre2 lhsmtool_posix: 1773685782.646494 [37851]: rebind [0x200000403:0x6a:0x0] to [0x200000403:0x6b:0x0] lhsmtool_posix: 1773685782.646638 [37851]: process finished, errs: 0 major, 0 minor, rc=0 (Success) PASS 14 (6s) == sanity-hsm test 15: Rebind a list of files ============ 14:29:45 (1773685785) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_15/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127761 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15758 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129872 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110761 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102766 s, 10.2 MB/s pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 0+0 records in 0+0 records out 0 bytes copied, 0.00211256 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00305491 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00258982 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00352873 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00412277 s, 0.0 kB/s rebind list of files lhsmtool_posix: 1773685789.693820 [38634]: action=2 src=/tmp/tmp.8346 dst=(null) mount_point=/mnt/lustre2 lhsmtool_posix: 1773685789.694903 [38634]: rebind [0x200000403:0x6e:0x0] to [0x200000403:0x73:0x0] lhsmtool_posix: 1773685789.695083 [38634]: rebind [0x200000403:0x6f:0x0] to [0x200000403:0x74:0x0] lhsmtool_posix: 1773685789.695216 [38634]: rebind [0x200000403:0x70:0x0] to [0x200000403:0x75:0x0] lhsmtool_posix: 1773685789.695261 [38634]: rebind [0x200000403:0x71:0x0] to [0x200000403:0x76:0x0] lhsmtool_posix: 1773685789.695306 [38634]: rebind [0x200000403:0x72:0x0] to [0x200000403:0x77:0x0] lhsmtool_posix: 1773685789.695371 [38634]: 5 lines read from '/tmp/tmp.8346', 5 rebind successful lhsmtool_posix: 1773685789.695473 [38634]: process finished, errs: 0 major, 0 minor, rc=0 (Success) PASS 15 (13s) == sanity-hsm test 16: Test CT bandwith control option === 14:29:58 (1773685798) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_16/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.045607 s, 460 MB/s Waiting 200s for 'SUCCEED' Waiting 190s for 'SUCCEED' Updated after 21s: want 'SUCCEED' got 'SUCCEED' PASS 16 (25s) == sanity-hsm test 20: Release is not permitted ========== 14:30:23 (1773685823) 0+0 records in 0+0 records out 0 bytes copied, 0.00258559 s, 0.0 kB/s 0x240000403:0x3f:0x0 Cannot send HSM request (use of /mnt/lustre/d20.sanity-hsm/f20.sanity-hsm): Operation not permitted Cannot send HSM request (use of /mnt/lustre/d20.sanity-hsm/f20.sanity-hsm): Operation not permitted Cannot send HSM request (use of /mnt/lustre/d20.sanity-hsm/f20.sanity-hsm): Operation not permitted Cannot send HSM request (use of /mnt/lustre/d20.sanity-hsm/f20.sanity-hsm): Operation not permitted PASS 20 (4s) == sanity-hsm test 21: Simple release tests ============== 14:30:27 (1773685827) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_21/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103346 s, 10.1 MB/s 1+0 records in 1+0 records out 512 bytes copied, 0.0560946 s, 9.1 kB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Waiting 200s for 'SUCCEED' PASS 21 (8s) == sanity-hsm test 22: Could not swap a release file ===== 14:30:35 (1773685835) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_22/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10557 s, 9.9 MB/s Waiting 200s for 'SUCCEED' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072202 s, 14.5 MB/s 0x200000403:0x81:0x0 lfs swap_layouts: error: cannot swap layout between '/mnt/lustre/d22.sanity-hsm/test_swap' and '/mnt/lustre/d22.sanity-hsm/test_release': Operation not supported (95) PASS 22 (5s) == sanity-hsm test 23: Release does not change a/mtime (utime) ========================================================== 14:30:40 (1773685840) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_23/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828296 s, 12.7 MB/s Waiting 200s for 'SUCCEED' PASS 23 (5s) == sanity-hsm test 24a: Archive, release, and restore does not change a/mtime (i/o) ========================================================== 14:30:45 (1773685845) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10273 s, 10.2 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Killing existing copytools on oleg434-client.virtnet copytools stopped in 0s 192.168.204.134@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg434-client.virtnet /mnt/lustre (opts:) Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre PASS 24a (8s) == sanity-hsm test 24b: root can archive, release, and restore user files ========================================================== 14:30:53 (1773685853) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102797 s, 10.2 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d24b.sanity-hsm/f24b.sanity-hsm] /mnt/lustre/d24b.sanity-hsm/f24b.sanity-hsm: (0x00000009) exists archived, archive_id:1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [md5sum] [/mnt/lustre/d24b.sanity-hsm/f24b.sanity-hsm] PASS 24b (7s) == sanity-hsm test 24c: check that user,group,other request masks work ========================================================== 14:31:01 (1773685861) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112096 s, 9.4 MB/s 0x200000403:0x8a:0x0 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_archive] [/mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm] Cannot send HSM request (use of /mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm): Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_archive] [/mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126613 s, 8.3 MB/s 0x200000403:0x8b:0x0 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_archive] [/mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm] Cannot send HSM request (use of /mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm): Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_archive] [/mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150726 s, 7.0 MB/s 0x200000403:0x8c:0x0 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_archive] [/mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm] Cannot send HSM request (use of /mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm): Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_archive] [/mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm] PASS 24c (11s) == sanity-hsm test 24d: check that read-only mounts are respected ========================================================== 14:31:12 (1773685872) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100242 s, 10.5 MB/s 0x200000403:0x8e:0x0 /mnt/lustre/d24d.sanity-hsm/f24d.sanity-hsm lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 75 0x4b 0x280000401 Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre3 Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre3"' Cannot send HSM request (use of /mnt/lustre2/d24d.sanity-hsm/f24d.sanity-hsm): Read-only file system Waiting 200s for 'SUCCEED' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 1.10413 s, 950 kB/s Cannot send HSM request (use of /mnt/lustre2/d24d.sanity-hsm/f24d.sanity-hsm): Read-only file system Stopping client oleg434-client.virtnet /mnt/lustre3 (opts:) PASS 24d (9s) == sanity-hsm test 24e: tar succeeds on HSM released files ========================================================== 14:31:21 (1773685881) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24e/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126332 s, 8.3 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' /mnt/lustre/d24e.sanity-hsm/f24e.sanity-hsm: (0x0000000d) released exists archived, archive_id:1 tar: Removing leading `/' from member names PASS 24e (7s) == sanity-hsm test 24f: root can archive, release, and restore tar files ========================================================== 14:31:28 (1773685888) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24f/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 54fb6627dbaa37721048e4549db3224d /mnt/lustre/d24f.sanity-hsm/f24f.sanity-hsm Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' f24f.sanity-hsm f24f.sanity-hsm Sum0 = 54fb6627dbaa37721048e4549db3224d /mnt/lustre/d24f.sanity-hsm/f24f.sanity-hsm, sum1 = 54fb6627dbaa37721048e4549db3224d /mnt/lustre/d24f.sanity-hsm/f24f.sanity-hsm PASS 24f (7s) == sanity-hsm test 24g: write by non-owner still sets dirty ========================================================== 14:31:35 (1773685895) RUNAS = 'runas -u 500 -g 500' Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24g/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Please listen carefully as our options have changed. Waiting 200s for 'SUCCEED' running as uid/gid/euid/egid 500/500/500/500, groups: 500 [tee] [/mnt/lustre/d24g.sanity-hsm/f24g.sanity-hsm] To be electrocuted by your telephone, press #. PASS 24g (5s) == sanity-hsm test 25a: Restore lost file (HS_LOST flag) from import ========================================================== 14:31:41 (1773685901) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_25a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1773685902.239625 [47723]: action=1 src=d25a.sanity-hsm/f25a.sanity-hsm dst=/mnt/lustre/d25a.sanity-hsm/f25a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1773685902.247265 [47723]: importing '/mnt/lustre/d25a.sanity-hsm/f25a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_25a//d25a.sanity-hsm/f25a.sanity-hsm' lhsmtool_posix: 1773685902.276623 [47723]: imported '/mnt/lustre/d25a.sanity-hsm/f25a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_25a//0406/0x240000404:0x2:0x0'=='/tmp/arc1/sanity-hsm.test_25a//d25a.sanity-hsm/f25a.sanity-hsm' lhsmtool_posix: 1773685902.276680 [47723]: process finished, errs: 0 major, 0 minor, rc=0 (Success) md5sum: /mnt/lustre/d25a.sanity-hsm/f25a.sanity-hsm: No data available PASS 25a (3s) == sanity-hsm test 25b: Restore lost file (HS_LOST flag) after release ========================================================== 14:31:44 (1773685904) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_25b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' md5sum: /mnt/lustre/d25b.sanity-hsm/f25b.sanity-hsm: No data available PASS 25b (6s) == sanity-hsm test 26A: Remove the archive of a valid file ========================================================== 14:31:50 (1773685910) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26A/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes copied, 0.00377071 s, 0.0 kB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' PASS 26A (8s) == sanity-hsm test 26a: Remove Archive On Last Unlink (RAoLU) policy ========================================================== 14:31:58 (1773685918) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Waiting 200s for 'SUCCEED' Updated after 5s: want 'SUCCEED' got 'SUCCEED' PASS 26a (22s) == sanity-hsm test 26b: RAoLU policy when CDT off ======== 14:32:21 (1773685941) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled Killing existing copytools on oleg434-client.virtnet copytools stopped in 0s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' PASS 26b (13s) == sanity-hsm test 26c: RAoLU effective when file closed ========================================================== 14:32:34 (1773685954) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' multiop /mnt/lustre/d26c.sanity-hsm/f26c.sanity-hsm vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8346 Waiting 200s for 'SUCCEED' Updated after 6s: want 'SUCCEED' got 'SUCCEED' Waiting 200s for 'SUCCEED' Updated after 8s: want 'SUCCEED' got 'SUCCEED' PASS 26c (30s) == sanity-hsm test 26d: RAoLU when Client eviction ======= 14:33:04 (1773685984) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104367 s, 10.0 MB/s multiop /mnt/lustre/d26d.sanity-hsm/f26d.sanity-hsm vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8346 Waiting 200s for 'SUCCEED' Updated after 8s: want 'SUCCEED' got 'SUCCEED' PASS 26d (22s) == sanity-hsm test 26e: RAoLU with a non-started coordinator ========================================================== 14:33:26 (1773686006) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26e/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143727 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152493 s, 6.9 MB/s Waiting 200s for 'SUCCEED' Killing existing copytools on oleg434-client.virtnet copytools stopped in 0s Failing mds1 on oleg434-server Stopping /mnt/lustre-mds1 (opts:) on oleg434-server 14:33:55 (1773686035) shut down facet: mds1 facet_host: oleg434-server facet_failover_host: oleg434-server Failover mds1 to oleg434-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0000 14:34:11 (1773686051) targets are mounted 14:34:11 (1773686051) facet_failover done oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Check cookie from RAoLU request (last: 1773685626, remove: 1773685627) mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26e/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' PASS 26e (83s) == sanity-hsm test 27a: Remove the archive of an imported file (Operation not permitted) ========================================================== 14:34:49 (1773686089) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_27a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 39+0 records in 39+0 records out 40894464 bytes (41 MB, 39 MiB) copied, 0.522593 s, 78.3 MB/s lhsmtool_posix: 1773686091.704129 [54905]: action=1 src=d27a.sanity-hsm/f27a.sanity-hsm dst=/mnt/lustre/d27a.sanity-hsm/f27a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1773686091.708708 [54905]: importing '/mnt/lustre/d27a.sanity-hsm/f27a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_27a//d27a.sanity-hsm/f27a.sanity-hsm' lhsmtool_posix: 1773686091.739596 [54905]: imported '/mnt/lustre/d27a.sanity-hsm/f27a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_27a//0404/0x200000407:0x3:0x0'=='/tmp/arc1/sanity-hsm.test_27a//d27a.sanity-hsm/f27a.sanity-hsm' lhsmtool_posix: 1773686091.739664 [54905]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Cannot send HSM request (use of /mnt/lustre/d27a.sanity-hsm/f27a.sanity-hsm): Operation not permitted PASS 27a (5s) == sanity-hsm test 27b: Remove the archive of a relased file (Operation not permitted) ========================================================== 14:34:54 (1773686094) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_27b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes copied, 0.00313143 s, 0.0 kB/s Waiting 200s for 'SUCCEED' Cannot send HSM request (use of /mnt/lustre/d27b.sanity-hsm/f27b.sanity-hsm): Operation not permitted PASS 27b (5s) == sanity-hsm test 28: Concurrent archive/file remove ==== 14:34:59 (1773686099) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_28/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes copied, 0.00696404 s, 0.0 kB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 28 (10s) == sanity-hsm test 29a: Tests --mntpath and --archive options ========================================================== 14:35:09 (1773686109) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_29a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 7 "/mnt/lustre"' Cannot send HSM request (use of /mnt/lustre): Invalid argument hsm: '/qwerty/uyt' is not a valid FID PASS 29a (3s) == sanity-hsm test 29b: Archive/delete/remove by FID from the archive. ========================================================== 14:35:12 (1773686112) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_29b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893278 s, 11.7 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' PASS 29b (7s) == sanity-hsm test 29c: Archive/delete/remove by FID, using a file list. ========================================================== 14:35:19 (1773686119) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_29c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122034 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840164 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103779 s, 10.1 MB/s Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' PASS 29c (10s) == sanity-hsm test 29d: hsm_remove by FID with archive_id 0 for unlinked file cause ========================================================== 14:35:29 (1773686129) SKIP: sanity-hsm test_29d Need 3 or more clients, have 1 SKIP 29d (1s) == sanity-hsm test 30a: Restore at exec (import case) ==== 14:35:31 (1773686131) SKIP: sanity-hsm test_30a Need 2 or more clients, have 1 SKIP 30a (1s) == sanity-hsm test 30b: Restore at exec (release case) === 14:35:33 (1773686133) SKIP: sanity-hsm test_30b Need 2 or more clients, have 1 SKIP 30b (3s) == sanity-hsm test 30c: Update during exec of released file must fail ========================================================== 14:35:35 (1773686135) SKIP: sanity-hsm test_30c Need 2 or more clients, have 1 SKIP 30c (2s) == sanity-hsm test 31a: Import a large file and check size during restore ========================================================== 14:35:37 (1773686137) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_31a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 39+0 records in 39+0 records out 40894464 bytes (41 MB, 39 MiB) copied, 0.517817 s, 79.0 MB/s lhsmtool_posix: 1773686138.880683 [60088]: action=1 src=d31a.sanity-hsm/f31a.sanity-hsm dst=/mnt/lustre/d31a.sanity-hsm/f31a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1773686138.888560 [60088]: importing '/mnt/lustre/d31a.sanity-hsm/f31a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_31a//d31a.sanity-hsm/f31a.sanity-hsm' lhsmtool_posix: 1773686138.923564 [60088]: imported '/mnt/lustre/d31a.sanity-hsm/f31a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_31a//040b/0x200000407:0xc:0x0'=='/tmp/arc1/sanity-hsm.test_31a//d31a.sanity-hsm/f31a.sanity-hsm' lhsmtool_posix: 1773686138.923810 [60088]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_31a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' size seen is right: 40894464 == 40894464 done PASS 31a (14s) == sanity-hsm test 31b: Restore a large unaligned file and check size during restore ========================================================== 14:35:51 (1773686151) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_31b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 39+0 records in 39+0 records out 39000000 bytes (39 MB, 37 MiB) copied, 0.434288 s, 89.8 MB/s Waiting 200s for 'SUCCEED' size seen is right: 39000000 == 39000000 done PASS 31b (16s) == sanity-hsm test 31c: Restore a large aligned file and check size during restore ========================================================== 14:36:08 (1773686168) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_31c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 39+0 records in 39+0 records out 40894464 bytes (41 MB, 39 MiB) copied, 0.439682 s, 93.0 MB/s Waiting 200s for 'SUCCEED' size seen is right: 40894464 == 40894464 done PASS 31c (16s) == sanity-hsm test 33: Kill a restore waiting process ==== 14:36:24 (1773686184) 0+0 records in 0+0 records out 0 bytes copied, 0.00559203 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_33/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Copytool is suspended on oleg434-client.virtnet Waiting 200s for 'STARTED' Copytool is continued on oleg434-client.virtnet /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh: line 2829: 62119 Terminated md5sum $f > /dev/null PASS 33 (7s) == sanity-hsm test 34: Remove file during restore ======== 14:36:31 (1773686191) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_34/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes copied, 0.00406924 s, 0.0 kB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Copytool is suspended on oleg434-client.virtnet Waiting 200s for 'STARTED' Updated after 2s: want 'STARTED' got 'STARTED' Copytool is continued on oleg434-client.virtnet /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh: line 2861: kill: (62795) - No such process PASS 34 (8s) == sanity-hsm test 35: Overwrite file during restore ===== 14:36:39 (1773686199) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_35/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes copied, 0.00236533 s, 0.0 kB/s Waiting 200s for 'SUCCEED' Copytool is suspended on oleg434-client.virtnet Waiting 200s for 'STARTED' Copytool is continued on oleg434-client.virtnet /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh: line 2898: kill: (63489) - No such process PASS 35 (7s) == sanity-hsm test 36: Move file during restore ========== 14:36:46 (1773686206) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_36/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes copied, 0.00350731 s, 0.0 kB/s Waiting 200s for 'SUCCEED' Copytool is suspended on oleg434-client.virtnet Copytool is continued on oleg434-client.virtnet /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh: line 2934: kill: (64180) - No such process PASS 36 (5s) == sanity-hsm test 37: re-archive a dirty file =========== 14:36:51 (1773686211) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_37/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663633 s, 15.8 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227763 s, 46.0 MB/s Waiting 200s for 'SUCCEED' Changed after 2s: from 'SUCCEED WAITING' to 'SUCCEED SUCCEED' Updated after 7s: want 'SUCCEED' got 'SUCCEED' PASS 37 (74s) == sanity-hsm test 40: Parallel archive requests ========= 14:38:05 (1773686285) oleg434-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/hsm/max_requests=1152921504606846976: Cannot allocate memory oleg434-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/max_requests'='1152921504606846976': Cannot allocate memory pdsh@oleg434-client: oleg434-server: ssh exited with exit code 12 oleg434-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0001/hsm/max_requests=1152921504606846976: Cannot allocate memory oleg434-server: error: set_param: setting 'mdt/lustre-MDT0001/hsm/max_requests'='1152921504606846976': Cannot allocate memory pdsh@oleg434-client: oleg434-server: ssh exited with exit code 12 set max_requests=1152921504606846976 failed with 12 [ 1072.048174] Lustre: lustre-MDT0000: No more memory to set HSM max_requests=1152921504606846976 (max request memory: 461MB, current total 6/975940): rc = -12 Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_40/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge Wait for all requests being enqueued...100 archive requests submitted 100 archive requests submitted 100 archive requests submitted 100 archive requests submitted OK pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 PASS 40 (103s) == sanity-hsm test 50: Archive with large number of pending HSM actions ========================================================== 14:39:48 (1773686388) oleg434-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/hsm/max_requests=1000000: Cannot allocate memory oleg434-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/max_requests'='1000000': Cannot allocate memory pdsh@oleg434-client: oleg434-server: ssh exited with exit code 12 oleg434-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0001/hsm/max_requests=1000000: Cannot allocate memory oleg434-server: error: set_param: setting 'mdt/lustre-MDT0001/hsm/max_requests'='1000000': Cannot allocate memory pdsh@oleg434-client: oleg434-server: ssh exited with exit code 12 Filesystem Inodes IUsed IFree IUse% Mounted on 192.168.204.134@tcp:/lustre 231603 1331 230272 1% /mnt/lustre - create 3043 (time 1773686401.48 total 10.00 last 304.28) - create 6479 (time 1773686411.48 total 20.00 last 343.59) total: 10000 create in 29.72 seconds: 336.50 ops/second 0 - unlinked 0 (time 1773686464 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second Start Phase files_num: 10000 time: 41 - create 3006 (time 1773686511.65 total 10.00 last 300.55) - create 6537 (time 1773686521.65 total 20.00 last 353.05) - create 10000 (time 1773686531.08 total 29.43 last 367.27) - create 13453 (time 1773686541.08 total 39.43 last 345.28) - create 17757 (time 1773686551.08 total 49.43 last 430.39) total: 20000 create in 54.02 seconds: 370.26 ops/second - unlinked 0 (time 1773686623 ; total 0 ; last 0) - unlinked 10000 (time 1773686648 ; total 25 ; last 25) total: 20000 unlinks in 47 seconds: 425.531921 unlinks/second Middle Phase files_num: 20000 time: 66 - create 5995 (time 1773686682.20 total 10.00 last 599.45) total: 10000 create in 18.57 seconds: 538.44 ops/second 0 - unlinked 0 (time 1773686726 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second End Phase files_num: 10000 time: 34 0 mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge PASS 50 (367s) == sanity-hsm test 52: Opened for write file on an evicted client should be set dirty ========================================================== 14:45:55 (1773686755) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_52/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725929 s, 14.4 MB/s multiop /mnt/lustre/d52.sanity-hsm/f52.sanity-hsm vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8346 pdsh@oleg434-client: oleg434-client: ssh exited with exit code 5 PASS 52 (6s) == sanity-hsm test 53: Opened for read file on an evicted client should not be set dirty ========================================================== 14:46:01 (1773686761) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_53/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683287 s, 15.3 MB/s Waiting 200s for 'SUCCEED' multiop /mnt/lustre/d53.sanity-hsm/f53.sanity-hsm vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.8346 pdsh@oleg434-client: oleg434-client: ssh exited with exit code 5 PASS 53 (8s) == sanity-hsm test 54: Write during an archive cancels it ========================================================== 14:46:09 (1773686769) 39+0 records in 39+0 records out 39000000 bytes (39 MB, 37 MiB) copied, 0.330758 s, 118 MB/s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_54/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' Waiting 200s for 'STARTED' mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0001.hsm.policy=+NRA Waiting 200s for 'FAILED' Waiting 190s for 'FAILED' Waiting 170s for 'FAILED' Updated after 37s: want 'FAILED' got 'FAILED' mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA PASS 54 (43s) == sanity-hsm test 55: Truncate during an archive cancels it ========================================================== 14:46:52 (1773686812) 39+0 records in 39+0 records out 39000000 bytes (39 MB, 37 MiB) copied, 0.356925 s, 109 MB/s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_55/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' Waiting 200s for 'STARTED' mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0001.hsm.policy=+NRA mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA PASS 55 (6s) == sanity-hsm test 56: Setattr during an archive is ok === 14:46:58 (1773686818) 39+0 records in 39+0 records out 39000000 bytes (39 MB, 37 MiB) copied, 0.356628 s, 109 MB/s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_56/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Waiting 190s for 'SUCCEED' Waiting 180s for 'SUCCEED' Updated after 36s: want 'SUCCEED' got 'SUCCEED' PASS 56 (40s) == sanity-hsm test 57: Archive a file with dirty cache on another node ========================================================== 14:47:38 (1773686858) SKIP: sanity-hsm test_57 Need 2 or more clients, have 1 SKIP 57 (1s) == sanity-hsm test 58: Truncate a released file will trigger restore ========================================================== 14:47:40 (1773686860) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_58/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' truncate up from 1516 to 3032 Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' /mnt/lustre/d58.sanity-hsm/f58.sanity-hsm: (0x0000000b) exists dirty archived, archive_id:2 truncate down from 1516 to 758 Waiting 200s for 'SUCCEED' /mnt/lustre/d58.sanity-hsm/f58.sanity-hsm: (0x0000000b) exists dirty archived, archive_id:2 truncate to 0 Waiting 200s for 'SUCCEED' /mnt/lustre/d58.sanity-hsm/f58.sanity-hsm: (0x0000000b) exists dirty archived, archive_id:2 PASS 58 (12s) == sanity-hsm test 59: Release stripeless file with non-zero size ========================================================== 14:47:51 (1773686871) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_59/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' PASS 59 (5s) == sanity-hsm test 60: Changing progress update interval from default ========================================================== 14:47:56 (1773686876) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.100994 s, 104 MB/s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_60/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 --update-interval 5 "/mnt/lustre2"' Waiting 100s for 'start copy' Expecting a progress update within 20 seconds... 5242880 bytes copied in 4 seconds. Wait for on going archive hsm action to complete Waiting 10s for 'copied' Updated after 4s: want 'copied' got 'copied' PASS 60 (14s) == sanity-hsm test 61: Waiting archive of a removed file should fail ========================================================== 14:48:10 (1773686890) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_61/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 61 (6s) == sanity-hsm test 70: Copytool logs JSON register/unregister events to FIFO ========================================================== 14:48:16 (1773686896) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_70/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --event-fifo /tmp/sanity-hsm.test_70.e5av/fifo "/mnt/lustre2"' Killing existing copytools on oleg434-client.virtnet copytools stopped in 0s Register/Unregister events look OK. PASS 70 (4s) == sanity-hsm test 71: Copytool logs JSON archive events to FIFO ========================================================== 14:48:20 (1773686900) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_71/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --update-interval 5 --event-fifo /tmp/sanity-hsm.test_71.XXMs/fifo "/mnt/lustre2"' mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723061 s, 14.5 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Archive events look OK. mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA PASS 71 (10s) == sanity-hsm test 72: Copytool logs JSON restore events to FIFO ========================================================== 14:48:30 (1773686910) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_72/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --update-interval 5 --event-fifo /tmp/sanity-hsm.test_72.X87S/fifo "/mnt/lustre2"' 16+0 records in 16+0 records out 16000000 bytes (16 MB, 15 MiB) copied, 0.173507 s, 92.2 MB/s lhsmtool_posix: 1773686912.440231 [82868]: action=1 src=d72.sanity-hsm/f72.sanity-hsm dst=/mnt/lustre/d72.sanity-hsm/f72.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1773686912.443426 [82868]: importing '/mnt/lustre/d72.sanity-hsm/f72.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_72//d72.sanity-hsm/f72.sanity-hsm' lhsmtool_posix: 1773686912.456759 [82868]: imported '/mnt/lustre/d72.sanity-hsm/f72.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_72//0bc5/0x200000bd2:0x17:0x0'=='/tmp/arc1/sanity-hsm.test_72//d72.sanity-hsm/f72.sanity-hsm' lhsmtool_posix: 1773686912.456804 [82868]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Verifying released state: Waiting 200s for 'SUCCEED' Restore events look OK. PASS 72 (7s) == sanity-hsm test 90: Archive/restore a file list ======= 14:48:37 (1773686917) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_90/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge Waiting 200s for '' pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Waiting 200s for '' Changed after 1s: from 'lrh=[type=10680000 len=136 idx=2/510] fid=[0x200000403:0x25b:0x0] dfid=[0x200000403:0x25b:0x0] compound/cookie=0x0/0x69b84d70 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[] lrh=[type=10680000 len=136 idx=2/511] fid=[0x200000403:0x25c:0x0] dfid=[0x200000403:0x25c:0x0] compound/cookie=0x0/0x69b84d71 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[] lrh=[type=10680000 len=136 idx=2/512] fid=[0x200000403:0x25d:0x0] dfid=[0x200000403:0x25d:0x0] compound/cookie=0x0/0x69b84d72 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[] lrh=[type=10680000 len=136 idx=2/513] fid=[0x200000403:0x25e:0x0] dfid=[0x200000403:0x25e:0x0] compound/cookie=0x0/0x69b84d73 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/514] fid=[0x200000403:0x25f:0x0] dfid=[0x200000403:0x25f:0x0] compound/cookie=0x0/0x69b84d74 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/515] fid=[0x200000403:0x260:0x0] dfid=[0x200000403:0x260:0x0] compound/cookie=0x0/0x69b84d75 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/516] fid=[0x200000403:0x261:0x0] dfid=[0x200000403:0x261:0x0] compound/cookie=0x0/0x69b84d76 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/517] fid=[0x200000403:0x262:0x0] dfid=[0x200000403:0x262:0x0] compound/cookie=0x0/0x69b84d77 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/518] fid=[0x200000403:0x263:0x0] dfid=[0x200000403:0x263:0x0] compound/cookie=0x0/0x69b84d78 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/519] fid=[0x200000403:0x264:0x0] dfid=[0x200000403:0x264:0x0] compound/cookie=0x0/0x69b84d79 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/520] fid=[0x200000403:0x265:0x0] dfid=[0x200000403:0x265:0x0] compound/cookie=0x0/0x69b84d7a action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/521] fid=[0x200000403:0x266:0x0] dfid=[0x200000403:0x266:0x0] compound/cookie=0x0/0x69b84d7b action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/522] fid=[0x200000403:0x267:0x0] dfid=[0x200000403:0x267:0x0] compound/cookie=0x0/0x69b84d7c action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/523] fid=[0x200000403:0x268:0x0] dfid=[0x200000403:0x268:0x0] compound/cookie=0x0/0x69b84d7d action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/524] fid=[0x200000403:0x269:0x0] dfid=[0x200000403:0x269:0x0] compound/cookie=0x0/0x69b84d7e action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/525] fid=[0x200000403:0x26a:0x0] dfid=[0x200000403:0x26a:0x0] compound/cookie=0x0/0x69b84d7f action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/526] fid=[0x200000403:0x26b:0x0] dfid=[0x200000403:0x26b:0x0] compound/cookie=0x0/0x69b84d80 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/527] fid=[0x200000403:0x26c:0x0] dfid=[0x200000403:0x26c:0x0] compound/cookie=0x0/0x69b84d81 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/528] fid=[0x200000403:0x26d:0x0] dfid=[0x200000403:0x26d:0x0] compound/cookie=0x0/0x69b84d82 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/529] fid=[0x200000403:0x26e:0x0] dfid=[0x200000403:0x26e:0x0] compound/cookie=0x0/0x69b84d83 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/530] fid=[0x200000403:0x26f:0x0] dfid=[0x200000403:0x26f:0x0] compound/cookie=0x0/0x69b84d84 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/531] fid=[0x200000403:0x270:0x0] dfid=[0x200000403:0x270:0x0] compound/cookie=0x0/0x69b84d85 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/532] fid=[0x200000403:0x271:0x0] dfid=[0x200000403:0x271:0x0] compound/cookie=0x0/0x69b84d86 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/533] fid=[0x200000403:0x272:0x0] dfid=[0x200000403:0x272:0x0] compound/cookie=0x0/0x69b84d87 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/534] fid=[0x200000403:0x273:0x0] dfid=[0x200000403:0x273:0x0] compound/cookie=0x0/0x69b84d88 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/535] fid=[0x200000403:0x274:0x0] dfid=[0x200000403:0x274:0x0] compound/cookie=0x0/0x69b84d89 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/536] fid=[0x200000403:0x275:0x0] dfid=[0x200000403:0x275:0x0] compound/cookie=0x0/0x69b84d8a action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/537] fid=[0x200000403:0x276:0x0] dfid=[0x200000403:0x276:0x0] compound/cookie=0x0/0x69b84d8b action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/538] fid=[0x200000403:0x277:0x0] dfid=[0x200000403:0x277:0x0] compound/cookie=0x0/0x69b84d8c action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/539] fid=[0x200000403:0x278:0x0] dfid=[0x200000403:0x278:0x0] compound/cookie=0x0/0x69b84d8d action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/540] fid=[0x200000403:0x279:0x0] dfid=[0x200000403:0x279:0x0] compound/cookie=0x0/0x69b84d8e action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/541] fid=[0x200000403:0x27a:0x0] dfid=[0x200000403:0x27a:0x0] compound/cookie=0x0/0x69b84d8f action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/542] fid=[0x200000403:0x27b:0x0] dfid=[0x200000403:0x27b:0x0] compound/cookie=0x0/0x69b84d90 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/543] fid=[0x200000403:0x27c:0x0] dfid=[0x200000403:0x27c:0x0] compound/cookie=0x0/0x69b84d91 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/544] fid=[0x200000403:0x27d:0x0] dfid=[0x200000403:0x27d:0x0] compound/cookie=0x0/0x69b84d92 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/545] fid=[0x200000403:0x27e:0x0] dfid=[0x200000403:0x27e:0x0] compound/cookie=0x0/0x69b84d93 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/546] fid=[0x200000403:0x27f:0x0] dfid=[0x200000403:0x27f:0x0] compound/cookie=0x0/0x69b84d94 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/547] fid=[0x200000403:0x280:0x0] dfid=[0x200000403:0x280:0x0] compound/cookie=0x0/0x69b84d95 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/548] fid=[0x200000403:0x281:0x0] dfid=[0x200000403:0x281:0x0] compound/cookie=0x0/0x69b84d96 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/550] fid=[0x200000403:0x282:0x0] dfid=[0x200000403:0x282:0x0] compound/cookie=0x0/0x69b84d97 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/551] fid=[0x200000403:0x283:0x0] dfid=[0x200000403:0x283:0x0] compound/cookie=0x0/0x69b84d98 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/552] fid=[0x200000403:0x284:0x0] dfid=[0x200000403:0x284:0x0] compound/cookie=0x0/0x69b84d99 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/553] fid=[0x200000403:0x285:0x0] dfid=[0x200000403:0x285:0x0] compound/cookie=0x0/0x69b84d9a action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/554] fid=[0x200000403:0x286:0x0] dfid=[0x200000403:0x286:0x0] compound/cookie=0x0/0x69b84d9b action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/555] fid=[0x200000403:0x287:0x0] dfid=[0x200000403:0x287:0x0] compound/cookie=0x0/0x69b84d9c action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[]' to 'lrh=[type=10680000 len=136 idx=2/550] fid=[0x200000403:0x282:0x0] dfid=[0x200000403:0x282:0x0] compound/cookie=0x0/0x69b84d97 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[] lrh=[type=10680000 len=136 idx=2/551] fid=[0x200000403:0x283:0x0] dfid=[0x200000403:0x283:0x0] compound/cookie=0x0/0x69b84d98 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[] lrh=[type=10680000 len=136 idx=2/552] fid=[0x200000403:0x284:0x0] dfid=[0x200000403:0x284:0x0] compound/cookie=0x0/0x69b84d99 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[] lrh=[type=10680000 len=136 idx=2/553] fid=[0x200000403:0x285:0x0] dfid=[0x200000403:0x285:0x0] compound/cookie=0x0/0x69b84d9a action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/554] fid=[0x200000403:0x286:0x0] dfid=[0x200000403:0x286:0x0] compound/cookie=0x0/0x69b84d9b action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=2/555] fid=[0x200000403:0x287:0x0] dfid=[0x200000403:0x287:0x0] compound/cookie=0x0/0x69b84d9c action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[]' pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 PASS 90 (75s) == sanity-hsm test 100: Set coordinator /proc tunables === 14:49:52 (1773686992) Testing mdt.lustre-MDT0000.hsm.loop_period Set loop_period to 0 must failed oleg434-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/hsm/loop_period=0: Invalid argument oleg434-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/loop_period'='0': Invalid argument pdsh@oleg434-client: oleg434-server: ssh exited with exit code 22 oleg434-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0001/hsm/loop_period=0: Invalid argument oleg434-server: error: set_param: setting 'mdt/lustre-MDT0001/hsm/loop_period'='0': Invalid argument pdsh@oleg434-client: oleg434-server: ssh exited with exit code 22 Testing mdt.lustre-MDT0000.hsm.grace_delay Set grace_delay to 0 must failed oleg434-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/hsm/grace_delay=0: Invalid argument oleg434-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/grace_delay'='0': Invalid argument pdsh@oleg434-client: oleg434-server: ssh exited with exit code 22 oleg434-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0001/hsm/grace_delay=0: Invalid argument oleg434-server: error: set_param: setting 'mdt/lustre-MDT0001/hsm/grace_delay'='0': Invalid argument pdsh@oleg434-client: oleg434-server: ssh exited with exit code 22 Testing mdt.lustre-MDT0000.hsm.active_request_timeout Set active_request_timeout to 0 must failed oleg434-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/hsm/active_request_timeout=0: Invalid argument oleg434-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/active_request_timeout'='0': Invalid argument pdsh@oleg434-client: oleg434-server: ssh exited with exit code 22 oleg434-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0001/hsm/active_request_timeout=0: Invalid argument oleg434-server: error: set_param: setting 'mdt/lustre-MDT0001/hsm/active_request_timeout'='0': Invalid argument pdsh@oleg434-client: oleg434-server: ssh exited with exit code 22 Testing mdt.lustre-MDT0000.hsm.max_requests Set max_requests to 0 must failed oleg434-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/hsm/max_requests=0: Invalid argument oleg434-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/max_requests'='0': Invalid argument pdsh@oleg434-client: oleg434-server: ssh exited with exit code 22 oleg434-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0001/hsm/max_requests=0: Invalid argument oleg434-server: error: set_param: setting 'mdt/lustre-MDT0001/hsm/max_requests'='0': Invalid argument pdsh@oleg434-client: oleg434-server: ssh exited with exit code 22 Testing mdt.lustre-MDT0000.hsm.default_archive_id Set default_archive_id to 0 must failed oleg434-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/hsm/default_archive_id=0: Invalid argument oleg434-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/default_archive_id'='0': Invalid argument pdsh@oleg434-client: oleg434-server: ssh exited with exit code 22 oleg434-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0001/hsm/default_archive_id=0: Invalid argument oleg434-server: error: set_param: setting 'mdt/lustre-MDT0001/hsm/default_archive_id'='0': Invalid argument pdsh@oleg434-client: oleg434-server: ssh exited with exit code 22 PASS 100 (20s) == sanity-hsm test 102: Verify coordinator control ======= 14:50:12 (1773687012) mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 102 (7s) == sanity-hsm test 103: Purge all requests =============== 14:50:19 (1773687019) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_103/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge Current requests pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 PASS 103 (6s) == sanity-hsm test 103a: Purge pending restore requests == 14:50:25 (1773687025) mdt.lustre-MDT0000.hsm.policy=-NBR mdt.lustre-MDT0001.hsm.policy=-NBR Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_103a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' (0x200000403:0x2bb:0x0|0x200000403:0x2bc:0x0|0x200000403:0x2bd:0x0|0x200000403:0x2be:0x0|0x200000403:0x2bf:0x0|0x200000403:0x2c0:0x0|0x200000403:0x2c1:0x0|0x200000403:0x2c2:0x0).*action=ARCHIVE.*status=SUCCEED Killing existing copytools on oleg434-client.virtnet copytools stopped in 0s cat: '/mnt/lustre/.lustre/fid/0x200000403:0x2bd:0x0': No data available cat: '/mnt/lustre/.lustre/fid/0x200000403:0x2bc:0x0': No data available cat: '/mnt/lustre/.lustre/fid/0x200000403:0x2bb:0x0': No data available cat: '/mnt/lustre/.lustre/fid/0x200000403:0x2be:0x0': No data available cat: '/mnt/lustre/.lustre/fid/0x200000403:0x2c0:0x0': No data available cat: '/mnt/lustre/.lustre/fid/0x200000403:0x2bf:0x0': No data available cat: '/mnt/lustre/.lustre/fid/0x200000403:0x2c1:0x0': No data available cat: '/mnt/lustre/.lustre/fid/0x200000403:0x2c2:0x0': No data available mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge PASS 103a (8s) == sanity-hsm test 104: Copy tool data field ============= 14:50:33 (1773687033) 0+0 records in 0+0 records out 0 bytes copied, 0.00214526 s, 0.0 kB/s mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge PASS 104 (6s) == sanity-hsm test 105: Restart of coordinator =========== 14:50:39 (1773687039) mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge PASS 105 (14s) == sanity-hsm test 106: Copytool register/unregister ===== 14:50:53 (1773687053) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_106/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' oleg434-client.virtnet: executing pgrep --pidfile=/var/run/lhsmtool_posix.pid --list-full hsmtool oleg434-client.virtnet: executing get_client_uuid /mnt/lustre2 found agent uuid=2242dc0c-b366-4b7b-bf7a-e0464872b03c archive_id=ANY requests=[current:0 ok:0 errors:0] on mds1 found agent uuid=2242dc0c-b366-4b7b-bf7a-e0464872b03c archive_id=ANY requests=[current:0 ok:0 errors:0] on mds2 oleg434-client.virtnet: 89552 Killing existing copytools on oleg434-client.virtnet copytools stopped in 0s pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 uuid not found in agent list on mds1 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 uuid not found in agent list on mds2 Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_106/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' oleg434-client.virtnet: executing pgrep --pidfile=/var/run/lhsmtool_posix.pid --list-full hsmtool oleg434-client.virtnet: executing get_client_uuid /mnt/lustre2 found agent uuid=2242dc0c-b366-4b7b-bf7a-e0464872b03c archive_id=ANY requests=[current:0 ok:0 errors:0] on mds1 found agent uuid=2242dc0c-b366-4b7b-bf7a-e0464872b03c archive_id=ANY requests=[current:0 ok:0 errors:0] on mds2 PASS 106 (10s) == sanity-hsm test 107: Copytool re-register after MDS restart ========================================================== 14:51:03 (1773687063) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_107/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Failing mds1 on oleg434-server Stopping /mnt/lustre-mds1 (opts:) on oleg434-server 14:51:07 (1773687067) shut down facet: mds1 facet_host: oleg434-server facet_failover_host: oleg434-server Failover mds1 to oleg434-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0000 14:51:21 (1773687081) targets are mounted 14:51:21 (1773687081) facet_failover done oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Waiting 200s for 'SUCCEED' PASS 107 (28s) == sanity-hsm test 109: Policy display/change ============ 14:51:31 (1773687091) mdt.lustre-MDT0000.hsm.policy=+NBR mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0000.hsm.policy=-NBR mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0000.hsm.policy=NRA NBR Next set_param must failed oleg434-server: error: set_param: setting /sys/kernel/debug/lustre/mdt/lustre-MDT0000/hsm/policy=wrong: Invalid argument oleg434-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/policy'='wrong': Invalid argument pdsh@oleg434-client: oleg434-server: ssh exited with exit code 22 Back to default policy mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0001.hsm.policy=+NRA mdt.lustre-MDT0000.hsm.policy=-NBR mdt.lustre-MDT0001.hsm.policy=-NBR PASS 109 (8s) == sanity-hsm test 110a: Non blocking restore policy (import case) ========================================================== 14:51:39 (1773687099) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_110a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1773687100.132928 [93414]: action=1 src=d110a.sanity-hsm/f110a.sanity-hsm dst=/mnt/lustre/d110a.sanity-hsm/f110a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1773687100.135442 [93414]: importing '/mnt/lustre/d110a.sanity-hsm/f110a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_110a//d110a.sanity-hsm/f110a.sanity-hsm' lhsmtool_posix: 1773687100.146438 [93414]: imported '/mnt/lustre/d110a.sanity-hsm/f110a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_110a//0bb1/0x200000bd2:0x63:0x0'=='/tmp/arc1/sanity-hsm.test_110a//d110a.sanity-hsm/f110a.sanity-hsm' lhsmtool_posix: 1773687100.146489 [93414]: process finished, errs: 0 major, 0 minor, rc=0 (Success) mdt.lustre-MDT0000.hsm.policy=+NBR mdt.lustre-MDT0001.hsm.policy=+NBR md5sum: /mnt/lustre/d110a.sanity-hsm/f110a.sanity-hsm: No data available Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' mdt.lustre-MDT0000.hsm.policy=-NBR mdt.lustre-MDT0001.hsm.policy=-NBR PASS 110a (5s) == sanity-hsm test 110b: Non blocking restore policy (release case) ========================================================== 14:51:44 (1773687104) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_110b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm.policy=+NBR mdt.lustre-MDT0001.hsm.policy=+NBR md5sum: /mnt/lustre/d110b.sanity-hsm/f110b.sanity-hsm: No data available mdt.lustre-MDT0000.hsm.policy=-NBR mdt.lustre-MDT0001.hsm.policy=-NBR PASS 110b (4s) == sanity-hsm test 111a: No retry policy (import case), restore will error ========================================================== 14:51:48 (1773687108) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_111a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1773687109.543854 [94809]: action=1 src=d111a.sanity-hsm/f111a.sanity-hsm dst=/mnt/lustre/d111a.sanity-hsm/f111a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1773687109.546676 [94809]: importing '/mnt/lustre/d111a.sanity-hsm/f111a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_111a//d111a.sanity-hsm/f111a.sanity-hsm' lhsmtool_posix: 1773687109.556236 [94809]: imported '/mnt/lustre/d111a.sanity-hsm/f111a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_111a//0bb5/0x200000bd2:0x67:0x0'=='/tmp/arc1/sanity-hsm.test_111a//d111a.sanity-hsm/f111a.sanity-hsm' lhsmtool_posix: 1773687109.556277 [94809]: process finished, errs: 0 major, 0 minor, rc=0 (Success) mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0001.hsm.policy=+NRA Remove from backend: 0x200000bd2:0x67:0x0 = /tmp/arc1/sanity-hsm.test_111a/0bb5/0x200000bd2:0x67:0x0 Waiting 200s for 'FAILED' mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA PASS 111a (6s) == sanity-hsm test 111b: No retry policy (release case), restore will error ========================================================== 14:51:54 (1773687114) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_111b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0001.hsm.policy=+NRA Waiting 200s for 'SUCCEED' Remove from backend: 0x200000403:0x2ca:0x0 = /tmp/arc1/sanity-hsm.test_111b/06c9/0x200000403:0x2ca:0x0 Waiting 200s for 'FAILED' Updated after 2s: want 'FAILED' got 'FAILED' mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA PASS 111b (7s) == sanity-hsm test 112: State of recorded request ======== 14:52:01 (1773687121) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_112/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled /mnt/lustre/d112.sanity-hsm/f112.sanity-hsm: ARCHIVE waiting (from 0 to EOF) mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 112 (6s) == sanity-hsm test 113: wrong stat after restore ========= 14:52:07 (1773687127) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02484 s, 42.2 MB/s Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre3 Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_113/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre3"' Waiting 200s for 'SUCCEED' Verifying released state: multiop /mnt/lustre/d113.sanity-hsm/f113.sanity-hsm voO_WRONLY:O_APPEND:_w4c TMPPIPE=/tmp/multiop_open_wait_pipe.8346 File: /mnt/lustre2/d113.sanity-hsm/f113.sanity-hsm Size: 1048576 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205306057421 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-16 14:52:07.000000000 -0400 Modify: 2026-03-16 14:52:07.000000000 -0400 Change: 2026-03-16 14:52:07.000000000 -0400 Birth: 2026-03-16 14:52:06.000000000 -0400 Stopping client oleg434-client.virtnet /mnt/lustre3 (opts:) PASS 113 (6s) == sanity-hsm test 114: Incompatible request does not set other requests as STARTED ========================================================== 14:52:13 (1773687133) 0+0 records in 0+0 records out 0 bytes copied, 0.00134593 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00115532 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_114/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 114 (6s) == sanity-hsm test 200: Register/Cancel archive ========== 14:52:19 (1773687139) 0+0 records in 0+0 records out 0 bytes copied, 0.00141377 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_200/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Copytool is suspended on oleg434-client.virtnet Waiting 200s for 'STARTED' Waiting 200s for 'CANCELED' Updated after 2s: want 'CANCELED' got 'CANCELED' Copytool is continued on oleg434-client.virtnet PASS 200 (6s) == sanity-hsm test 201: Register/Cancel restore ========== 14:52:25 (1773687145) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_201/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 39+0 records in 39+0 records out 40894464 bytes (41 MB, 39 MiB) copied, 0.429817 s, 95.1 MB/s lhsmtool_posix: 1773687146.908253 [99235]: action=1 src=d201.sanity-hsm/f201.sanity-hsm dst=/mnt/lustre/d201.sanity-hsm/f201.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1773687146.911480 [99235]: importing '/mnt/lustre/d201.sanity-hsm/f201.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_201//d201.sanity-hsm/f201.sanity-hsm' lhsmtool_posix: 1773687146.931162 [99235]: imported '/mnt/lustre/d201.sanity-hsm/f201.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_201//0ba3/0x200000bd2:0x71:0x0'=='/tmp/arc1/sanity-hsm.test_201//d201.sanity-hsm/f201.sanity-hsm' lhsmtool_posix: 1773687146.931226 [99235]: process finished, errs: 0 major, 0 minor, rc=0 (Success) mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 201 (7s) == sanity-hsm test 202: Register/Cancel remove =========== 14:52:32 (1773687152) 0+0 records in 0+0 records out 0 bytes copied, 0.00202817 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_202/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Copytool is suspended on oleg434-client.virtnet Waiting 200s for 'STARTED' PASS 202 (6s) == sanity-hsm test 220A: Changelog for archive =========== 14:52:39 (1773687159) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_220A/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' lustre-MDT0000.3 16HSM 18:52:41.077052333 2026.03.16 0x0 t=[0x200000403:0x2d4:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 PASS 220A (8s) == sanity-hsm test 220a: Changelog for failed archive ==== 14:52:46 (1773687166) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_220a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl2 cl2' Copytool is suspended on oleg434-client.virtnet Waiting 200s for 'STARTED' Copytool is continued on oleg434-client.virtnet lustre-MDT0000.6 16HSM 18:52:50.482234577 2026.03.16 0x2 t=[0x200000403:0x2d5:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp p=[0x200000bd2:0x74:0x0] lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0001: Deregistered changelog user #2 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 PASS 220a (9s) == sanity-hsm test 221: Changelog for archive canceled === 14:52:55 (1773687175) 0+0 records in 0+0 records out 0 bytes copied, 0.00179099 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_221/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl3 cl3' Copytool is suspended on oleg434-client.virtnet Waiting 200s for 'CANCELED' Updated after 2s: want 'CANCELED' got 'CANCELED' Copytool is continued on oleg434-client.virtnet lustre-MDT0000.8 16HSM 18:53:00.558178847 2026.03.16 0x7d t=[0x200000403:0x2d6:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp lustre-MDT0001: clear the changelog for cl3 of all records lustre-MDT0001: Deregistered changelog user #3 lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0000: Deregistered changelog user #3 PASS 221 (11s) == sanity-hsm test 222a: Changelog for explicit restore == 14:53:06 (1773687186) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_222a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1773687187.650875 [103099]: action=1 src=d222a.sanity-hsm/f222a.sanity-hsm dst=/mnt/lustre/d222a.sanity-hsm/f222a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1773687187.656586 [103099]: importing '/mnt/lustre/d222a.sanity-hsm/f222a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_222a//d222a.sanity-hsm/f222a.sanity-hsm' lhsmtool_posix: 1773687187.675435 [103099]: imported '/mnt/lustre/d222a.sanity-hsm/f222a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_222a//0ba5/0x200000bd2:0x77:0x0'=='/tmp/arc1/sanity-hsm.test_222a//d222a.sanity-hsm/f222a.sanity-hsm' lhsmtool_posix: 1773687187.675512 [103099]: process finished, errs: 0 major, 0 minor, rc=0 (Success) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl4 cl4' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' lustre-MDT0000.11 16HSM 18:53:09.946743941 2026.03.16 0x80 t=[0x200000bd2:0x77:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp lustre-MDT0001: clear the changelog for cl4 of all records lustre-MDT0001: Deregistered changelog user #4 lustre-MDT0000: clear the changelog for cl4 of all records lustre-MDT0000: Deregistered changelog user #4 PASS 222a (9s) == sanity-hsm test 222b: Changelog for implicit restore == 14:53:15 (1773687195) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_222b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl5 cl5' Waiting 200s for 'SUCCEED' 4ef076ad20e5d4a2f4729d6ed41a3bf9 /mnt/lustre/d222b.sanity-hsm/f222b.sanity-hsm lustre-MDT0000.20 16HSM 18:53:19.333031929 2026.03.16 0x80 t=[0x200000403:0x2d8:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp p=[0x200000bd2:0x74:0x0] lustre-MDT0001: clear the changelog for cl5 of all records lustre-MDT0001: Deregistered changelog user #5 lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0000: Deregistered changelog user #5 PASS 222b (9s) == sanity-hsm test 222c: Changelog for failed explicit restore ========================================================== 14:53:24 (1773687204) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_222c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1773687205.036729 [104741]: action=1 src=d222c.sanity-hsm/f222c.sanity-hsm dst=/mnt/lustre/d222c.sanity-hsm/f222c.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1773687205.039518 [104741]: importing '/mnt/lustre/d222c.sanity-hsm/f222c.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_222c//d222c.sanity-hsm/f222c.sanity-hsm' lhsmtool_posix: 1773687205.049670 [104741]: imported '/mnt/lustre/d222c.sanity-hsm/f222c.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_222c//0ba9/0x200000bd2:0x7b:0x0'=='/tmp/arc1/sanity-hsm.test_222c//d222c.sanity-hsm/f222c.sanity-hsm' lhsmtool_posix: 1773687205.049714 [104741]: process finished, errs: 0 major, 0 minor, rc=0 (Success) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl6 cl6' Copytool is suspended on oleg434-client.virtnet Copytool is continued on oleg434-client.virtnet lustre-MDT0000.22 16HSM 18:53:26.637159216 2026.03.16 0x82 t=[0x200000bd2:0x7b:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp p=[0x200000bd2:0x7a:0x0] lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0001: Deregistered changelog user #6 lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0000: Deregistered changelog user #6 PASS 222c (7s) == sanity-hsm test 222d: Changelog for failed implicit restore ========================================================== 14:53:31 (1773687211) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_222d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl7 cl7' Waiting 200s for 'SUCCEED' Remove from backend: 0x200000403:0x2da:0x0 = /tmp/arc1/sanity-hsm.test_222d/06d9/0x200000403:0x2da:0x0 md5sum: /mnt/lustre/d222d.sanity-hsm/f222d.sanity-hsm: No data available lustre-MDT0000.29 16HSM 18:53:35.863451457 2026.03.16 0x82 t=[0x200000403:0x2da:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp lustre-MDT0001: clear the changelog for cl7 of all records lustre-MDT0001: Deregistered changelog user #7 lustre-MDT0000: clear the changelog for cl7 of all records lustre-MDT0000: Deregistered changelog user #7 PASS 222d (10s) == sanity-hsm test 223a: Changelog for restore canceled (import case) ========================================================== 14:53:41 (1773687221) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_223a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' 39+0 records in 39+0 records out 40894464 bytes (41 MB, 39 MiB) copied, 0.431949 s, 94.7 MB/s mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl8 cl8' lhsmtool_posix: 1773687224.396335 [106487]: action=1 src=d223a.sanity-hsm/f223a.sanity-hsm dst=/mnt/lustre/d223a.sanity-hsm/f223a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1773687224.399380 [106487]: importing '/mnt/lustre/d223a.sanity-hsm/f223a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_223a//d223a.sanity-hsm/f223a.sanity-hsm' lhsmtool_posix: 1773687224.413454 [106487]: imported '/mnt/lustre/d223a.sanity-hsm/f223a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_223a//0b52/0x200000bd2:0x80:0x0'=='/tmp/arc1/sanity-hsm.test_223a//d223a.sanity-hsm/f223a.sanity-hsm' lhsmtool_posix: 1773687224.413519 [106487]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Waiting 200s for 'STARTED' Waiting 200s for 'CANCELED' Waiting 200s for 'SUCCEED' Waiting 190s for 'SUCCEED' Waiting 180s for 'SUCCEED' Updated after 28s: want 'SUCCEED' got 'SUCCEED' lustre-MDT0000.36 16HSM 18:54:14.424687958 2026.03.16 0xfd t=[0x200000bd2:0x80:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp p=[0x200000bd2:0x7a:0x0] lustre-MDT0001: clear the changelog for cl8 of all records lustre-MDT0001: Deregistered changelog user #8 lustre-MDT0000: clear the changelog for cl8 of all records lustre-MDT0000: Deregistered changelog user #8 PASS 223a (38s) == sanity-hsm test 223b: Changelog for restore canceled (release case) ========================================================== 14:54:19 (1773687259) 0+0 records in 0+0 records out 0 bytes copied, 0.00136992 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_223b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl9 cl9' Copytool is suspended on oleg434-client.virtnet Waiting 200s for 'STARTED' Updated after 2s: want 'STARTED' got 'STARTED' Copytool is continued on oleg434-client.virtnet lustre-MDT0000.43 16HSM 18:54:24.017765778 2026.03.16 0xfd t=[0x200000403:0x2dd:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp p=[0x200000bd2:0x7a:0x0] lustre-MDT0001: clear the changelog for cl9 of all records lustre-MDT0001: Deregistered changelog user #9 lustre-MDT0000: clear the changelog for cl9 of all records lustre-MDT0000: Deregistered changelog user #9 PASS 223b (9s) == sanity-hsm test 224A: Changelog for remove ============ 14:54:29 (1773687269) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_224A/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl10 cl10' Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' lustre-MDT0000.48 16HSM 18:54:33.249343045 2026.03.16 0x200 t=[0x200000403:0x2df:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp lustre-MDT0001: clear the changelog for cl10 of all records lustre-MDT0001: Deregistered changelog user #10 lustre-MDT0000: clear the changelog for cl10 of all records lustre-MDT0000: Deregistered changelog user #10 PASS 224A (9s) == sanity-hsm test 224a: Changelog for failed remove ===== 14:54:38 (1773687278) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_224a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl11 cl11' Waiting 200s for 'SUCCEED' Remove from backend: 0x200000403:0x2e0:0x0 = /tmp/arc1/sanity-hsm.test_224a/06e3/0x200000403:0x2e0:0x0 Copytool is suspended on oleg434-client.virtnet Waiting 200s for 'STARTED' Copytool is continued on oleg434-client.virtnet lustre-MDT0000.53 16HSM 18:54:43.792450655 2026.03.16 0x202 t=[0x200000403:0x2e0:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp p=[0x200000bd2:0x84:0x0] lustre-MDT0001: clear the changelog for cl11 of all records lustre-MDT0001: Deregistered changelog user #11 lustre-MDT0000: clear the changelog for cl11 of all records lustre-MDT0000: Deregistered changelog user #11 PASS 224a (11s) == sanity-hsm test 225: Changelog for remove canceled ==== 14:54:49 (1773687289) Test disabled PASS 225 (2s) == sanity-hsm test 226: changelog for last rm/mv with exiting archive ========================================================== 14:54:51 (1773687291) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_226/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 0x200000403:0x2e3:0x0 mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl12 cl12' Waiting 200s for 'SUCCEED' lustre-MDT0000.54 16HSM 18:54:54.043881732 2026.03.16 0x280 t=[0x200000403:0x2e1:0x0] ef=0x13 u=0:0 nid=0@<0:0> lustre-MDT0000.55 16HSM 18:54:54.070300493 2026.03.16 0x280 t=[0x200000403:0x2e1:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp lustre-MDT0000.56 16HSM 18:54:54.070452233 2026.03.16 0x0 t=[0x200000403:0x2e1:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp lustre-MDT0000.57 16HSM 18:54:55.131818612 2026.03.16 0x280 t=[0x200000403:0x2e2:0x0] ef=0x13 u=0:0 nid=0@<0:0> lustre-MDT0000.58 16HSM 18:54:55.151628646 2026.03.16 0x280 t=[0x200000403:0x2e2:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp lustre-MDT0000.59 16HSM 18:54:55.151771001 2026.03.16 0x0 t=[0x200000403:0x2e2:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp lustre-MDT0000.60 06UNLNK 18:54:55.311682532 2026.03.16 0x3 t=[0x200000403:0x2e1:0x0] j=rm.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp p=[0x200000bd2:0x85:0x0] f226.sanity-hsm-1 lustre-MDT0000.60 06UNLNK 18:54:55.311682532 2026.03.16 0x3 t=[0x200000403:0x2e1:0x0] j=rm.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp p=[0x200000bd2:0x85:0x0] f226.sanity-hsm-1 lustre-MDT0000.61 08RENME 18:54:55.465128701 2026.03.16 0x3 t=[0x200000403:0x2e2:0x0] j=mv.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp p=[0x200000bd2:0x85:0x0] f226.sanity-hsm-2 s=[0x200000403:0x2e3:0x0] sp=[0x200000bd2:0x85:0x0] f226.sanity-hsm-3 lustre-MDT0001: clear the changelog for cl12 of all records lustre-MDT0001: Deregistered changelog user #12 lustre-MDT0000: clear the changelog for cl12 of all records lustre-MDT0000: Deregistered changelog user #12 PASS 226 (8s) == sanity-hsm test 227: changelog when explicit setting of HSM flags ========================================================== 14:54:59 (1773687299) 0+0 records in 0+0 records out 0 bytes copied, 0.00155025 s, 0.0 kB/s mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl13 cl13' lustre-MDT0001: clear the changelog for cl13 of all records lustre-MDT0001: Deregistered changelog user #13 lustre-MDT0000: clear the changelog for cl13 of all records lustre-MDT0000: Deregistered changelog user #13 PASS 227 (8s) == sanity-hsm test 228: On released file, return extend to FIEMAP. For [cp,tar] --sparse ========================================================== 14:55:07 (1773687307) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_228/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209252 s, 50.1 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' /mnt/lustre/f228.sanity-hsm: 1 extent found tar: Removing leading `/' from member names mnt/lustre/f228.sanity-hsm PASS 228 (6s) == sanity-hsm test 250: Coordinator max request ========== 14:55:13 (1773687313) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_250/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes copied, 0.00151585 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00554438 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00166847 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00146094 s, 0.0 kB/s Waiting 200s for 'SUCCEED' 0+0 records in 0+0 records out 0 bytes copied, 0.00186637 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00123771 s, 0.0 kB/s Waiting 200s for 'SUCCEED' Copytool is suspended on oleg434-client.virtnet Waiting 200s for 'STARTED' Copytool is continued on oleg434-client.virtnet PASS 250 (14s) == sanity-hsm test 251: Coordinator request timeout ====== 14:55:27 (1773687327) 0+0 records in 0+0 records out 0 bytes copied, 0.00145754 s, 0.0 kB/s mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_251/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Copytool is suspended on oleg434-client.virtnet Waiting 200s for 'STARTED' Updated after 3s: want 'STARTED' got 'STARTED' Waiting 200s for 'CANCELED' PASS 251 (13s) == sanity-hsm test 252: Timeout'ed running archive of a removed file should be canceled ========================================================== 14:55:41 (1773687341) 0+0 records in 0+0 records out 0 bytes copied, 0.00131596 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_252/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Copytool is suspended on oleg434-client.virtnet Waiting 200s for 'CANCELED' Updated after 2s: want 'CANCELED' got 'CANCELED' Copytool is continued on oleg434-client.virtnet PASS 252 (7s) == sanity-hsm test 253: Check for wrong file size after release ========================================================== 14:55:48 (1773687348) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_253/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 10+0 records in 10+0 records out 10000000 bytes (10 MB, 9.5 MiB) copied, 0.0793298 s, 126 MB/s Waiting 200s for 'SUCCEED' fail_loc=0x807 Cannot send HSM request (use of /mnt/lustre/d253.sanity-hsm/f253.sanity-hsm): Invalid argument could not release file PASS 253 (4s) == sanity-hsm test 254a: Request counters are initialized to zero ========================================================== 14:55:52 (1773687352) PASS 254a (3s) == sanity-hsm test 254b: Request counters are correctly incremented and decremented ========================================================== 14:55:55 (1773687355) Will launch 59 requests of each type Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_254b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Checking archive requests Copytool is suspended on oleg434-client.virtnet Copytool is continued on oleg434-client.virtnet Checking restore requests Copytool is suspended on oleg434-client.virtnet Copytool is continued on oleg434-client.virtnet Checking remove requests Copytool is suspended on oleg434-client.virtnet Copytool is continued on oleg434-client.virtnet PASS 254b (20s) == sanity-hsm test 255: Copytool registration wakes the coordinator up ========================================================== 14:56:15 (1773687375) 0+0 records in 0+0 records out 0 bytes copied, 0.00155477 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_255/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Killing existing copytools on oleg434-client.virtnet copytools stopped in 0s 0+0 records in 0+0 records out 0 bytes copied, 0.00364419 s, 0.0 kB/s 0x200000403:0x36c:0x0 mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_255/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' PASS 255 (13s) == sanity-hsm test 260a: Restore request have priority over other requests ========================================================== 14:56:29 (1773687389) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515394 s, 20.3 MB/s 0x200000403:0x36e:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639695 s, 16.4 MB/s 0x200000403:0x36f:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539785 s, 19.4 MB/s 0x200000403:0x370:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589875 s, 17.8 MB/s 0x200000403:0x371:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063077 s, 16.6 MB/s 0x200000403:0x372:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061339 s, 17.1 MB/s 0x200000403:0x373:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682766 s, 15.4 MB/s 0x200000403:0x374:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557237 s, 18.8 MB/s 0x200000403:0x375:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626709 s, 16.7 MB/s 0x200000403:0x376:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616427 s, 17.0 MB/s 0x200000403:0x377:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607899 s, 17.2 MB/s 0x200000403:0x378:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524374 s, 20.0 MB/s 0x200000403:0x379:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532029 s, 19.7 MB/s 0x200000403:0x37a:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665958 s, 15.7 MB/s 0x200000403:0x37b:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544362 s, 19.3 MB/s 0x200000403:0x37c:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587545 s, 17.8 MB/s 0x200000403:0x37d:0x0 Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Killing existing copytools on oleg434-client.virtnet copytools stopped in 0s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' ARCHIVE ARCHIVE RESTORE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE PASS 260a (15s) == sanity-hsm test 260b: Restore request have priority over other requests ========================================================== 14:56:44 (1773687404) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633658 s, 16.5 MB/s 0x200000403:0x380:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608749 s, 17.2 MB/s 0x200000403:0x381:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602599 s, 17.4 MB/s 0x200000403:0x382:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780898 s, 13.4 MB/s 0x200000403:0x383:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543794 s, 19.3 MB/s 0x200000403:0x384:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076417 s, 13.7 MB/s 0x200000403:0x385:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064242 s, 16.3 MB/s 0x200000403:0x386:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564669 s, 18.6 MB/s 0x200000403:0x387:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626402 s, 16.7 MB/s 0x200000403:0x388:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656081 s, 16.0 MB/s 0x200000403:0x389:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560115 s, 18.7 MB/s 0x200000403:0x38a:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644598 s, 16.3 MB/s 0x200000403:0x38b:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696183 s, 15.1 MB/s 0x200000403:0x38c:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592052 s, 17.7 MB/s 0x200000403:0x38d:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541839 s, 19.4 MB/s 0x200000403:0x38e:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667283 s, 15.7 MB/s 0x200000403:0x38f:0x0 Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2 "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Killing existing copytools on oleg434-client.virtnet copytools stopped in 0s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2 "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' RESTORE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE PASS 260b (16s) == sanity-hsm test 260c: Requests are not reordered on the 'hot' path of the coordinator ========================================================== 14:57:00 (1773687420) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455223 s, 23.0 MB/s 0x200000403:0x392:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561386 s, 18.7 MB/s 0x200000403:0x393:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523119 s, 20.0 MB/s 0x200000403:0x394:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554069 s, 18.9 MB/s 0x200000403:0x395:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490488 s, 21.4 MB/s 0x200000403:0x396:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049744 s, 21.1 MB/s 0x200000403:0x397:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495999 s, 21.1 MB/s 0x200000403:0x398:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498621 s, 21.0 MB/s 0x200000403:0x399:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651262 s, 16.1 MB/s 0x200000403:0x39a:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529386 s, 19.8 MB/s 0x200000403:0x39b:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583362 s, 18.0 MB/s 0x200000403:0x39c:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490135 s, 21.4 MB/s 0x200000403:0x39d:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555804 s, 18.9 MB/s 0x200000403:0x39e:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575845 s, 18.2 MB/s 0x200000403:0x39f:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511213 s, 20.5 MB/s 0x200000403:0x3a0:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565948 s, 18.5 MB/s 0x200000403:0x3a1:0x0 Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2 "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Killing existing copytools on oleg434-client.virtnet copytools stopped in 0s mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2 "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE RESTORE PASS 260c (23s) == sanity-hsm test 261: Report 0 bytes size after HSM release ========================================================== 14:57:23 (1773687443) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_261/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00610657 s, 1.3 MB/s /mnt/lustre/d261.sanity-hsm/f261.sanity-hsm: (0x00000009) exists archived, archive_id:2 Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' /mnt/lustre/d261.sanity-hsm/f261.sanity-hsm: (0x0000000d) released exists archived, archive_id:2 PASS 261 (5s) == sanity-hsm test 262: The client should return 1 block for HSM released files ========================================================== 14:57:28 (1773687448) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_262/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00560079 s, 1.5 MB/s Waiting 200s for 'SUCCEED' PASS 262 (5s) == sanity-hsm test 300: On disk coordinator state kept between MDT umount/mount ========================================================== 14:57:33 (1773687453) Stop coordinator and remove coordinator state at mount mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown Failing mds1 on oleg434-server Stopping /mnt/lustre-mds1 (opts:) on oleg434-server 14:57:38 (1773687458) shut down facet: mds1 facet_host: oleg434-server facet_failover_host: oleg434-server Failover mds1 to oleg434-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0000 14:57:53 (1773687473) targets are mounted 14:57:53 (1773687473) facet_failover done oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Set coordinator start at mount, and start coordinator Failing mds1 on oleg434-server Stopping /mnt/lustre-mds1 (opts:) on oleg434-server 14:58:24 (1773687504) shut down facet: mds1 facet_host: oleg434-server facet_failover_host: oleg434-server Failover mds1 to oleg434-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0000 14:58:39 (1773687519) targets are mounted 14:58:39 (1773687519) facet_failover done oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 300 (75s) == sanity-hsm test 301: HSM tunnable are persistent ====== 14:58:48 (1773687528) Failing mds1 on oleg434-server Stopping /mnt/lustre-mds1 (opts:) on oleg434-server 14:58:51 (1773687531) shut down facet: mds1 facet_host: oleg434-server facet_failover_host: oleg434-server Failover mds1 to oleg434-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0000 14:59:06 (1773687546) targets are mounted 14:59:06 (1773687546) facet_failover done oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 301 (27s) == sanity-hsm test 302: HSM tunnable are persistent when CDT is off ========================================================== 14:59:15 (1773687555) mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown Failing mds1 on oleg434-server Stopping /mnt/lustre-mds1 (opts:) on oleg434-server 14:59:20 (1773687560) shut down facet: mds1 facet_host: oleg434-server facet_failover_host: oleg434-server Failover mds1 to oleg434-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0000 14:59:34 (1773687574) targets are mounted 14:59:34 (1773687574) facet_failover done oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg434-server Stopping /mnt/lustre-mds2 (opts:) on oleg434-server 14:59:42 (1773687582) shut down facet: mds2 facet_host: oleg434-server facet_failover_host: oleg434-server Failover mds2 to oleg434-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0001 14:59:56 (1773687596) targets are mounted 14:59:56 (1773687596) facet_failover done oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 302 (51s) == sanity-hsm test 400: Single request is sent to the right MDT ========================================================== 15:00:06 (1773687606) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_400/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681298 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602361 s, 17.4 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' archive successful on mdt0 Waiting 200s for 'SUCCEED' archive successful on mdt1 PASS 400 (7s) == sanity-hsm test 401: Compound requests split and sent to their respective MDTs ========================================================== 15:00:13 (1773687613) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_401/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629558 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658568 s, 15.9 MB/s archive successful on mdt0 Waiting 200s for 'SUCCEED' archive successful on mdt1 PASS 401 (4s) == sanity-hsm test 402a: Copytool start fails if all MDTs are inactive ========================================================== 15:00:18 (1773687618) deactivate lustre-MDT0000-mdc-ffff917ac4bd1800 on oleg434-client.virtnet deactivate lustre-MDT0001-mdc-ffff917ac4bd1800 on oleg434-client.virtnet deactivate lustre-MDT0000-mdc-ffff917ac9c5d800 on oleg434-client.virtnet deactivate lustre-MDT0001-mdc-ffff917ac9c5d800 on oleg434-client.virtnet Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_402a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Failed to start copytool agt1 on 'oleg434-client.virtnet' pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 uuid not found in agent list on mds1 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 uuid not found in agent list on mds2 activate lustre-MDT0000-mdc-ffff917ac4bd1800 on oleg434-client.virtnet activate lustre-MDT0001-mdc-ffff917ac4bd1800 on oleg434-client.virtnet activate lustre-MDT0000-mdc-ffff917ac9c5d800 on oleg434-client.virtnet activate lustre-MDT0001-mdc-ffff917ac9c5d800 on oleg434-client.virtnet PASS 402a (3s) == sanity-hsm test 402b: CDT must retry request upon slow start of CT ========================================================== 15:00:21 (1773687621) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_402b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' fail_loc=0x14d fail_loc=0 Waiting 200s for 'SUCCEED' Updated after 3s: want 'SUCCEED' got 'SUCCEED' PASS 402b (16s) == sanity-hsm test 403: Copytool starts with inactive MDT and register on reconnect ========================================================== 15:00:37 (1773687637) deactivate lustre-MDT0001-mdc-ffff917ac4bd1800 on oleg434-client.virtnet deactivate lustre-MDT0001-mdc-ffff917ac9c5d800 on oleg434-client.virtnet Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_403/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' oleg434-client.virtnet: executing pgrep --pidfile=/var/run/lhsmtool_posix.pid --list-full hsmtool oleg434-client.virtnet: executing get_client_uuid /mnt/lustre2 found agent uuid=2242dc0c-b366-4b7b-bf7a-e0464872b03c archive_id=ANY requests=[current:0 ok:0 errors:0] on mds1 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 uuid not found in agent list on mds2 oleg434-client.virtnet: 133059 activate lustre-MDT0001-mdc-ffff917ac4bd1800 on oleg434-client.virtnet activate lustre-MDT0001-mdc-ffff917ac9c5d800 on oleg434-client.virtnet found agent uuid=2242dc0c-b366-4b7b-bf7a-e0464872b03c archive_id=ANY requests=[current:0 ok:0 errors:0] on mds1 found agent uuid=2242dc0c-b366-4b7b-bf7a-e0464872b03c archive_id=ANY requests=[current:0 ok:0 errors:0] on mds2 PASS 403 (6s) == sanity-hsm test 404: Inactive MDT does not block requests for active MDTs ========================================================== 15:00:43 (1773687643) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_404/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643587 s, 16.3 MB/s deactivate lustre-MDT0001-mdc-ffff917ac4bd1800 on oleg434-client.virtnet deactivate lustre-MDT0001-mdc-ffff917ac9c5d800 on oleg434-client.virtnet Waiting 200s for 'SUCCEED' archive successful on mdt0 activate lustre-MDT0001-mdc-ffff917ac4bd1800 on oleg434-client.virtnet activate lustre-MDT0001-mdc-ffff917ac9c5d800 on oleg434-client.virtnet PASS 404 (5s) == sanity-hsm test 405: archive and release under striped directory ========================================================== 15:00:48 (1773687648) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_405/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209918 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210134 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266434 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251269 s, 41.7 MB/s Waiting 200s for 'SUCCEED' archive successful on 0x240000bd2:0x1:0x0 archive successful on 0x240000bd2:0x2:0x0 archive successful on 0x240000bd2:0x3:0x0 archive successful on 0x2000032e3:0x6:0x0 PASS 405 (9s) == sanity-hsm test 406: attempting to migrate HSM archived files is safe ========================================================== 15:00:57 (1773687657) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568678 s, 18.4 MB/s old fid 0x2000032e3:0x9:0x0 Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_406/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' lfs migrate: /mnt/lustre/d406.sanity-hsm/f406.sanity-hsm migrate failed: Operation not supported (95) lfs migrate: cb_migrate_mdt_fini: error completing migration of /mnt/lustre/d406.sanity-hsm: Directory not empty (39) Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Waiting 200s for 'SUCCEED' new fid 0x240000bd3:0x7:0x0 Waiting 200s for 'SUCCEED' debug=+trace Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' PASS 406 (11s) == sanity-hsm test 407: Check for double RESTORE records in llog ========================================================== 15:01:08 (1773687668) 0+0 records in 0+0 records out 0 bytes copied, 0.00189506 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_407/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' fail_val=5 fail_loc=0x164 Copytool is suspended on oleg434-client.virtnet mdt.lustre-MDT0000.hsm.actions= lrh=[type=10680000 len=136 idx=4/1] fid=[0x200000403:0x3a8:0x0] dfid=[0x200000403:0x3a8:0x0] compound/cookie=0x0/0x69b85315 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=4/2] fid=[0x200000403:0x3ac:0x0] dfid=[0x200000403:0x3ac:0x0] compound/cookie=0x0/0x69b85316 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=4/3] fid=[0x2000032e2:0x2:0x0] dfid=[0x2000032e2:0x2:0x0] compound/cookie=0x0/0x69b85317 action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=4/4] fid=[0x2000032e3:0x3:0x0] dfid=[0x2000032e3:0x3:0x0] compound/cookie=0x0/0x69b85318 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=4/5] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x6:0x0] compound/cookie=0x0/0x69b85319 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=4/6] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x7:0x0] compound/cookie=0x0/0x69b8531a action=RESTORE archive#=3 flags=0x4 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=4/7] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x69b8531b action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=4/8] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0xa:0x0] compound/cookie=0x0/0x69b8531c action=RESTORE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=4/9] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x69b8531d action=REMOVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=4/10] fid=[0x2000032e3:0xc:0x0] dfid=[0x2000032e3:0xc:0x0] compound/cookie=0x0/0x69b8531e action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] Failing mds1 on oleg434-server Stopping /mnt/lustre-mds1 (opts:) on oleg434-server 15:01:17 (1773687677) shut down facet: mds1 facet_host: oleg434-server facet_failover_host: oleg434-server Failover mds1 to oleg434-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0000 15:01:31 (1773687691) targets are mounted 15:01:31 (1773687691) facet_failover done oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 mdt.lustre-MDT0000.hsm.actions= lrh=[type=10680000 len=136 idx=4/3] fid=[0x2000032e2:0x2:0x0] dfid=[0x2000032e2:0x2:0x0] compound/cookie=0x0/0x69b85317 action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=4/4] fid=[0x2000032e3:0x3:0x0] dfid=[0x2000032e3:0x3:0x0] compound/cookie=0x0/0x69b85318 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=4/5] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x6:0x0] compound/cookie=0x0/0x69b85319 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=4/6] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x7:0x0] compound/cookie=0x0/0x69b8531a action=RESTORE archive#=3 flags=0x4 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=4/7] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x69b8531b action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=4/8] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0xa:0x0] compound/cookie=0x0/0x69b8531c action=RESTORE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=4/9] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x69b8531d action=REMOVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=4/10] fid=[0x2000032e3:0xc:0x0] dfid=[0x2000032e3:0xc:0x0] compound/cookie=0x0/0x69b8531e action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=4/11] fid=[0x2000032e3:0xc:0x0] dfid=[0x2000032e3:0xc:0x0] compound/cookie=0x0/0x69b8531f action=RESTORE archive#=2 flags=0x4 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[] Copytool is continued on oleg434-client.virtnet d41d8cd98f00b204e9800998ecf8427e /mnt/lustre/d407.sanity-hsm/f407.sanity-hsm d41d8cd98f00b204e9800998ecf8427e /mnt/lustre2/d407.sanity-hsm/f407.sanity-hsm pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 mdt.lustre-MDT0000.hsm.actions= lrh=[type=10680000 len=136 idx=4/4] fid=[0x2000032e3:0x3:0x0] dfid=[0x2000032e3:0x3:0x0] compound/cookie=0x0/0x69b85318 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=4/5] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x6:0x0] compound/cookie=0x0/0x69b85319 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=4/6] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x7:0x0] compound/cookie=0x0/0x69b8531a action=RESTORE archive#=3 flags=0x4 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=4/7] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x69b8531b action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=4/8] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0xa:0x0] compound/cookie=0x0/0x69b8531c action=RESTORE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=4/9] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x69b8531d action=REMOVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=4/10] fid=[0x2000032e3:0xc:0x0] dfid=[0x2000032e3:0xc:0x0] compound/cookie=0x0/0x69b8531e action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=4/11] fid=[0x2000032e3:0xc:0x0] dfid=[0x2000032e3:0xd:0x0] compound/cookie=0x0/0x69b8531f action=RESTORE archive#=2 flags=0x4 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] PASS 407 (35s) == sanity-hsm test 408: Verify fiemap on release file ==== 15:01:43 (1773687703) Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_408/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0351196 s, 1.9 MB/s SKIP: sanity-hsm test_408 ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 408 (2s) == sanity-hsm test 409a: Coordinator should not stop when in use ========================================================== 15:01:45 (1773687705) 0+0 records in 0+0 records out 0 bytes copied, 0.00161758 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_409a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' fail_val=5 fail_loc=0x164 cat: /mnt/lustre/d409a.sanity-hsm/f409a.sanity-hsm: No data available mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled Killing existing copytools on oleg434-client.virtnet copytools stopped in 0s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_409a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 409a (16s) == sanity-hsm test 409b: getattr released file with CDT stopped after remount ========================================================== 15:02:01 (1773687721) 0+0 records in 0+0 records out 0 bytes copied, 0.0020771 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_409b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Killing existing copytools on oleg434-client.virtnet copytools stopped in 0s Failing mds1 on oleg434-server Stopping /mnt/lustre-mds1 (opts:) on oleg434-server 15:02:26 (1773687746) shut down facet: mds1 facet_host: oleg434-server facet_failover_host: oleg434-server Failover mds1 to oleg434-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0000 15:02:40 (1773687760) targets are mounted 15:02:40 (1773687760) facet_failover done oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec File: /mnt/lustre/d409b.sanity-hsm/f409b.sanity-hsm Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115406632648720 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-16 15:02:01.000000000 -0400 Modify: 2026-03-16 15:02:01.000000000 -0400 Change: 2026-03-16 15:02:01.000000000 -0400 Birth: 2026-03-16 15:02:00.000000000 -0400 PASS 409b (68s) == sanity-hsm test 410: lfs data_version -s allows release of force-archived file ========================================================== 15:03:09 (1773687789) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812823 s, 12.9 MB/s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_410/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Cannot send HSM request (use of /mnt/lustre/d410.sanity-hsm/f410.sanity-hsm): Operation not permitted PASS 410 (2s) == sanity-hsm test 411: hsm_ops rbac role ================ 15:03:11 (1773687791) On MGS 192.168.204.134, active = nodemap.active=1 On MGS 192.168.204.134, test_411.trusted_nodemap = nodemap.test_411.trusted_nodemap=1 Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_411/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' /mnt/lustre/d411.sanity-hsm/fileA: (0x00000000) Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' Killing existing copytools on oleg434-client.virtnet copytools stopped in 0s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_411/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' On MGS 192.168.204.134, test_411.rbac = nodemap.test_411.rbac=file_perms,dne_ops,quota_ops,byfid_ops,chlg_ops,fscrypt_admin,server_upcall,ignore_root_prjquota,local_admin,pool_quota_ops /mnt/lustre/d411.sanity-hsm/fileB: (0x00000000) Cannot send HSM request (use of /mnt/lustre/d411.sanity-hsm/fileB): Permission denied Cannot send HSM request (use of /mnt/lustre/d411.sanity-hsm/fileB): Operation not permitted Cannot send HSM request (use of /mnt/lustre/d411.sanity-hsm/fileB): Permission denied Cannot send HSM request (use of /mnt/lustre/d411.sanity-hsm/fileB): Permission denied lfs hsm_set: change hsm flags for '/mnt/lustre/d411.sanity-hsm/fileB' failed: Permission denied lfs hsm_clear: change hsm flags for '/mnt/lustre/d411.sanity-hsm/fileB' failed: Permission denied On MGS 192.168.204.134, active = nodemap.active=0 oleg434-server: error: nodemap 'test_411' does not exist oleg434-server: nodemap_del: Invalid argument pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 On MGS 192.168.204.134, active = nodemap.active=0 PASS 411 (17s) == sanity-hsm test 500: various LLAPI HSM tests ========== 15:03:28 (1773687808) striped dir -i0 -c2 -H crush /mnt/lustre/d500.sanity-hsm test1 @1773687828: Register/unregister copytool 2000x to check for leaks ...... pass test2 @1773687828: Re/un-register copytool multiple times without error ....... pass test3 @1773687828: Pass bad parameters to llapi_hsm_copytool_register() ....... pass test4 @1773687828: Bad parameters to llapi_hsm_copytool_unregister() .......... pass test5 @1773687828: Test llapi_hsm_copytool_recv() in non-blocking mode ........ pass test6 @1773687828: Test llapi_hsm_copytool_recv() with bogus parameters ....... pass test7 @1773687828: Test event polling (without actual traffic) ................ pass test50 @1773687828: Test llapi_hsm_state_get()/get_fd() ....................... pass test51 @1773687828: Test llapi_hsm_state_set_fd() ............................. pass test52 @1773687828: Test llapi_hsm_current_action() ........................... pass test100 @1773687829: Simple archive creation with no progress reported. ....... pass test101 @1773687831: Simple archive creation with progress every byte. ........ pass test102 @1773687833: Archive creation with progress every byte, backwards ..... pass test103 @1773687834: Archive creation with a single progress report ........... pass test104 @1773687835: Archive creation with two progress reports ............... pass test105 @1773687836: Archive creation with one bogus progress report .......... pass test106 @1773687837: Archive creation with one empty progress report .......... pass test107 @1773687838: Archive creation with one bogus progress report .......... pass test108 @1773687840: Archive creation with same progress report each time ..... pass test109 @1773687841: Archive creation with one report, with large number ...... pass test110 @1773687842: Archive with 10 progress reports, checking progress ...... pass test111 @1773687843: Archive with 10 reports in reverse, checking progress .... pass test112 @1773687844: Archive with 10 reports, duplicated, check progress ...... pass test113 @1773687846: Archive with 9 reports, with 20% overlapping coverage .... pass PASS 500 (39s) == sanity-hsm test 600: Changelog fields 'u=' and 'nid=' ========================================================== 15:04:07 (1773687847) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d600.sanity-hsm/f600.sanity-hsm] Got NID '192.168.204.34@tcp' lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 PASS 600 (7s) == sanity-hsm test 601: OPEN Changelog entry ============= 15:04:14 (1773687854) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl2 cl2' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0001.4 10OPEN 19:04:16.813973085 2026.03.16 0x1 t=[0x240000bd3:0x1b:0x0] j=cat.0 ef=0x17 u=0:0 nid=192.168.204.34@tcp m=r-- p=[0x2000032e2:0x19:0x0] lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0001: Deregistered changelog user #2 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 PASS 601 (7s) == sanity-hsm test 602: Changelog record CLOSE only if open+write or OPEN recorded ========================================================== 15:04:21 (1773687861) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl3 cl3' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0001: clear the changelog for cl3 of all records lustre-MDT0000.9 11CLOSE 19:04:23.228258540 2026.03.16 0x1 t=[0x2000032e2:0x1c:0x0] j=cat.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0001: clear the changelog for cl3 of all records lustre-MDT0000.10 10OPEN 19:04:23.342230980 2026.03.16 0x242 t=[0x2000032e2:0x1c:0x0] j=bash.0 ef=0x17 u=0:0 nid=192.168.204.34@tcp m=-w- p=[0x2000032e2:0x19:0x0] lustre-MDT0000.11 13TRUNC 19:04:23.344975531 2026.03.16 0xe t=[0x2000032e2:0x1c:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp p=[0x2000032e2:0x1b:0x0] lustre-MDT0000.12 11CLOSE 19:04:23.352416779 2026.03.16 0x242 t=[0x2000032e2:0x1c:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp lustre-MDT0000.12 11CLOSE 19:04:23.352416779 2026.03.16 0x242 t=[0x2000032e2:0x1c:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp mdd.lustre-MDT0000.changelog_mask=-OPEN mdd.lustre-MDT0001.changelog_mask=-OPEN lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0001: clear the changelog for cl3 of all records f lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0001: clear the changelog for cl3 of all records lustre-MDT0000.13 13TRUNC 19:04:24.035880519 2026.03.16 0xe t=[0x2000032e2:0x1c:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp p=[0x2000032e2:0x1b:0x0] lustre-MDT0000.14 11CLOSE 19:04:24.039857740 2026.03.16 0x242 t=[0x2000032e2:0x1c:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp lustre-MDT0000.14 11CLOSE 19:04:24.039857740 2026.03.16 0x242 t=[0x2000032e2:0x1c:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp lustre-MDT0001: clear the changelog for cl3 of all records lustre-MDT0001: Deregistered changelog user #3 lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0000: Deregistered changelog user #3 PASS 602 (7s) == sanity-hsm test 603: GETXATTR Changelog entry ========= 15:04:28 (1773687868) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl4 cl4' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL lustre-MDT0000: clear the changelog for cl4 of all records lustre-MDT0001: clear the changelog for cl4 of all records getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/d603.sanity-hsm/f603.sanity-hsm user.xattr1="value1" lustre-MDT0001.10 23GXATR 19:04:30.349421463 2026.03.16 0x0 t=[0x240000bd3:0x1d:0x0] j=getfattr.0 ef=0x1b u=0:0 nid=192.168.204.34@tcp x=user.xattr1 lustre-MDT0001: clear the changelog for cl4 of all records lustre-MDT0001: Deregistered changelog user #4 lustre-MDT0000: clear the changelog for cl4 of all records lustre-MDT0000: Deregistered changelog user #4 PASS 603 (6s) == sanity-hsm test 604: NOPEN Changelog entry ============ 15:04:34 (1773687874) mdd.lustre-MDT0000.changelog_deniednext=20 mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl5 cl5' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0001: clear the changelog for cl5 of all records running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cat] [/mnt/lustre2/d604.sanity-hsm/f604.sanity-hsm] cat: /mnt/lustre2/d604.sanity-hsm/f604.sanity-hsm: Permission denied lustre-MDT0000.19 24NOPEN 19:04:36.954755027 2026.03.16 0x1 t=[0x2000032e2:0x1e:0x0] j=cat.500 ef=0x17 u=500:500 nid=192.168.204.34@tcp m=r-- Got NID '192.168.204.34@tcp' lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0001: clear the changelog for cl5 of all records running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cat] [/mnt/lustre2/d604.sanity-hsm/f604.sanity-hsm] cat: /mnt/lustre2/d604.sanity-hsm/f604.sanity-hsm: Permission denied lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0001: clear the changelog for cl5 of all records running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cat] [/mnt/lustre2/d604.sanity-hsm/f604.sanity-hsm] cat: /mnt/lustre2/d604.sanity-hsm/f604.sanity-hsm: Permission denied lustre-MDT0000.20 24NOPEN 19:04:57.261506514 2026.03.16 0x1 t=[0x2000032e2:0x1e:0x0] j=cat.500 ef=0x17 u=500:500 nid=192.168.204.34@tcp m=r-- p=[0x2000032e2:0x1b:0x0] Got NID '192.168.204.34@tcp' lustre-MDT0001: clear the changelog for cl5 of all records lustre-MDT0001: Deregistered changelog user #5 lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0000: Deregistered changelog user #5 mdd.lustre-MDT0000.changelog_deniednext=60 PASS 604 (28s) == sanity-hsm test 605: Test OPEN and CLOSE rate limit in Changelogs ========================================================== 15:05:02 (1773687902) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl6 cl6' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0000.24 10OPEN 19:05:04.086857751 2026.03.16 0x43 t=[0x2000032e2:0x20:0x0] j=bash.0 ef=0x17 u=0:0 nid=192.168.204.34@tcp m=rw- lustre-MDT0000.24 10OPEN 19:05:04.086857751 2026.03.16 0x43 t=[0x2000032e2:0x20:0x0] j=bash.0 ef=0x17 u=0:0 nid=192.168.204.34@tcp m=rw- lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0000.25 10OPEN 19:05:04.453110667 2026.03.16 0x1 t=[0x2000032e2:0x20:0x0] j=cat.0 ef=0x17 u=0:0 nid=192.168.204.34@tcp m=r-- p=[0x2000032e2:0x1f:0x0] lustre-MDT0000.26 11CLOSE 19:05:04.459817002 2026.03.16 0x1 t=[0x2000032e2:0x20:0x0] j=cat.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp lustre-MDT0000.25 10OPEN 19:05:04.453110667 2026.03.16 0x1 t=[0x2000032e2:0x20:0x0] j=cat.0 ef=0x17 u=0:0 nid=192.168.204.34@tcp m=r-- p=[0x2000032e2:0x1f:0x0] lustre-MDT0000.26 11CLOSE 19:05:04.459817002 2026.03.16 0x1 t=[0x2000032e2:0x20:0x0] j=cat.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0001: clear the changelog for cl6 of all records running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cat] [/mnt/lustre/d605.sanity-hsm/f605.sanity-hsm] lustre-MDT0000.27 10OPEN 19:05:04.867514999 2026.03.16 0x1 t=[0x2000032e2:0x20:0x0] j=cat.500 ef=0x17 u=500:500 nid=192.168.204.34@tcp m=r-- p=[0x2000032e2:0x1f:0x0] lustre-MDT0000.28 11CLOSE 19:05:04.871029283 2026.03.16 0x1 t=[0x2000032e2:0x20:0x0] j=cat.500 ef=0x13 u=500:500 nid=192.168.204.34@tcp lustre-MDT0000.27 10OPEN 19:05:04.867514999 2026.03.16 0x1 t=[0x2000032e2:0x20:0x0] j=cat.500 ef=0x17 u=500:500 nid=192.168.204.34@tcp m=r-- p=[0x2000032e2:0x1f:0x0] lustre-MDT0000.28 11CLOSE 19:05:04.871029283 2026.03.16 0x1 t=[0x2000032e2:0x20:0x0] j=cat.500 ef=0x13 u=500:500 nid=192.168.204.34@tcp lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0000.29 11CLOSE 19:05:05.066243957 2026.03.16 0x43 t=[0x2000032e2:0x20:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp lustre-MDT0000.29 11CLOSE 19:05:05.066243957 2026.03.16 0x43 t=[0x2000032e2:0x20:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.204.34@tcp lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0001: Deregistered changelog user #6 lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0000: Deregistered changelog user #6 PASS 605 (7s) == sanity-hsm test 606: llog_reader groks changelog fields ========================================================== 15:05:09 (1773687909) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl7 cl7' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d606.sanity-hsm/f606.sanity-hsm] Stopping /mnt/lustre-mds1 (opts:) on oleg434-server rec #2 type=1064553b len=64 offset 8256 Header size : 8192 llh_size : 64 Time : Mon Mar 16 14:51:18 2026 Number of records: 1 cat_idx: 0 last_idx: 2 Flags: 0x2012 Target uuid : ----------------------- #02 (064)fid=[0x1:0x3a:0x0] path=oi.1/0x1:0x3a:0x0 Got UID/GID 500:500 Got NID 192.168.204.34@tcp Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0000 lustre-MDT0001: clear the changelog for cl7 of all records lustre-MDT0001: Deregistered changelog user #7 lustre-MDT0000: clear the changelog for cl7 of all records lustre-MDT0000: Deregistered changelog user #7 PASS 606 (19s) == sanity-hsm test 607a: release a file that was migrated after being archived ========================================================== 15:05:28 (1773687928) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777409 s, 13.5 MB/s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_607a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' PASS 607a (6s) == sanity-hsm test 607b: Migrate should not change the HSM attribute of dirty files ========================================================== 15:05:34 (1773687934) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086429 s, 12.1 MB/s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_607b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' llite.lustre-ffff917ac4bd1800.xattr_cache=0 llite.lustre-ffff917ac9c5d800.xattr_cache=0 /mnt/lustre/d607b.sanity-hsm/f607b.sanity-hsm: (0x0000000b) exists dirty archived, archive_id:1 llite.lustre-ffff917ac4bd1800.xattr_cache=1 llite.lustre-ffff917ac9c5d800.xattr_cache=1 PASS 607b (4s) == sanity-hsm test 607c: 'lfs swap_layouts' should set dirty flag on HSM file ========================================================== 15:05:38 (1773687938) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920827 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839426 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819578 s, 12.8 MB/s Starting copytool 'agt1' on 'oleg434-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_607c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' lfs swap_layouts: error: cannot swap layout between '/mnt/lustre/d607c.sanity-hsm/f607c.sanity-hsm-1' and '/mnt/lustre/d607c.sanity-hsm/f607c.sanity-hsm-3': Operation not permitted (1) PASS 607c (5s) == sanity-hsm test complete, duration 2510 sec =========== 15:05:43 (1773687943) === sanity-hsm: start cleanup 15:05:44 (1773687944) === Stopping clients: oleg434-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg434-client.virtnet /mnt/lustre2 opts: === sanity-hsm: finish cleanup 15:05:54 (1773687954) ===