-----============= acceptance-small: sanity-hsm ============----- Sun Jun 7 00:51:32 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg406-server: /home/green/git/lustre-release/lustre/tests/except/sanity-hsm.ex - see MDS1_VERSION <= CLIENT_VERSION (34682169 <= 34682169) for LU-18924, go 100 105 excepting tests: === sanity-hsm: start setup 00:51:40 (1780807900) === Starting client oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre2 Mount client oleg406-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre2 Started clients oleg406-client.virtnet: 192.168.204.106@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg406-client.virtnet: executing check_config_client /mnt/lustre oleg406-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg406-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff90f1831cd800.idle_timeout=debug osc.lustre-OST0000-osc-ffff90f198813800.idle_timeout=debug osc.lustre-OST0001-osc-ffff90f1831cd800.idle_timeout=debug osc.lustre-OST0001-osc-ffff90f198813800.idle_timeout=debug disable quota as required oleg406-server: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-hsm: finish setup 00:51:55 (1780807915) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8369] nobody:x:65534: Killing existing copytools on oleg406-client.virtnet with signal TERM pkill: pidfile not valid Try `pkill --help' for more information. Set HSM on and start Set sanity-hsm HSM policy == sanity-hsm test 1A: lfs hsm flags root/non-root access ========================================================== 00:52:19 (1780807939) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_set] [--norelease] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_clear] [--norelease] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_set] [--exists] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] lfs hsm_set: change hsm flags for '/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm' failed: Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] PASS 1A (3s) == sanity-hsm test 1a: mmap & cat a HSM released file ==== 00:52:22 (1780807942) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125595 s, 8.3 MB/s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_1a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Verifying released state: PASS 1a (7s) == sanity-hsm test 1b: Archive, Release and Restore composite file ========================================================== 00:52:29 (1780807949) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354881 s, 29.5 MB/s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_1b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' archive /mnt/lustre/d1b.sanity-hsm/f1b.sanity-hsm Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' release /mnt/lustre/d1b.sanity-hsm/f1b.sanity-hsm verify released state: pass restore /mnt/lustre/d1b.sanity-hsm/f1b.sanity-hsm Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' verify restored state: pass PASS 1b (7s) == sanity-hsm test 1c: Check setting archive-id in lfs hsm_set ========================================================== 00:52:36 (1780807956) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d1c.sanity-hsm/f1c.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1c.sanity-hsm/f1c.sanity-hsm] verifying archive number is 32 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1c.sanity-hsm/f1c.sanity-hsm] verifying archive number is still 32 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1c.sanity-hsm/f1c.sanity-hsm] verifying archive number is 33 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1c.sanity-hsm/f1c.sanity-hsm] verifying archive number is 16 PASS 1c (4s) == sanity-hsm test 1d: Archive, Release and Restore DoM file ========================================================== 00:52:40 (1780807960) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387383 s, 27.1 MB/s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_1d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' archive /mnt/lustre/d1d.sanity-hsm/f1d.sanity-hsm Waiting 200s for 'SUCCEED' release /mnt/lustre/d1d.sanity-hsm/f1d.sanity-hsm verify released state: pass restore /mnt/lustre/d1d.sanity-hsm/f1d.sanity-hsm Waiting 200s for 'SUCCEED' verify restored state: pass PASS 1d (7s) == sanity-hsm test 1e: Archive, Release and Restore SEL file ========================================================== 00:52:47 (1780807967) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264067 s, 39.7 MB/s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_1e/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' archive /mnt/lustre/d1e.sanity-hsm/f1e.sanity-hsm Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' release /mnt/lustre/d1e.sanity-hsm/f1e.sanity-hsm verify released state: pass restore /mnt/lustre/d1e.sanity-hsm/f1e.sanity-hsm Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' verify restored state: pass PASS 1e (8s) == sanity-hsm test 1f: DoM file release after restore ==== 00:52:55 (1780807975) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451994 s, 23.2 MB/s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_1f/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' archive /mnt/lustre/d1f.sanity-hsm/f1f.sanity-hsm Waiting 200s for 'SUCCEED' release /mnt/lustre/d1f.sanity-hsm/f1f.sanity-hsm verify released state: pass restore /mnt/lustre/d1f.sanity-hsm/f1f.sanity-hsm Waiting 200s for 'SUCCEED' verify restored state: pass release again /mnt/lustre/d1f.sanity-hsm/f1f.sanity-hsm /mnt/lustre/d1f.sanity-hsm/f1f.sanity-hsm: (0x0000000d) released exists archived, archive_id:1 verify released state: pass PASS 1f (7s) == sanity-hsm test 2: Check file dirtyness when doing setattr ========================================================== 00:53:02 (1780807982) 0+0 records in 0+0 records out 0 bytes copied, 0.00497425 s, 0.0 kB/s 0x200000403:0x8:0x0 PASS 2 (4s) == sanity-hsm test 3: Check file dirtyness when opening for write ========================================================== 00:53:06 (1780807986) Succeed in opening file "/mnt/lustre/d3.sanity-hsm/f3.sanity-hsm"(flags=O_WRONLY) 3+0 records in 3+0 records out 3+0 records in 3+0 records out PASS 3 (4s) == sanity-hsm test 4: Useless cancel must not be registered ========================================================== 00:53:10 (1780807990) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108178 s, 9.7 MB/s PASS 4 (4s) == sanity-hsm test 8: Test default archive number ======== 00:53:14 (1780807994) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_8/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' PASS 8 (5s) == sanity-hsm test 9A: Use of explicit archive number, with dedicated copytool ========================================================== 00:53:19 (1780807999) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_9A/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 3 "/mnt/lustre2"' oleg406-client.virtnet: executing pgrep --pidfile=/var/run/lhsmtool_posix.pid --list-full hsmtool oleg406-client.virtnet: executing get_client_uuid /mnt/lustre2 found agent uuid=c90ca412-7c0e-430e-8b00-175241015fe3 archive_id=3 requests=[current:0 ok:0 errors:0] on mds1 found agent uuid=c90ca412-7c0e-430e-8b00-175241015fe3 archive_id=3 requests=[current:0 ok:0 errors:0] on mds2 Waiting 200s for 'SUCCEED' PASS 9A (13s) == sanity-hsm test 9a: Multiple remote agents ============ 00:53:32 (1780808012) SKIP: sanity-hsm test_9a Need 3 or more clients, have 1 SKIP 9a (2s) == sanity-hsm test 10a: Archive a file =================== 00:53:34 (1780808014) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_10a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh: line 996: warning: command substitution: ignored null byte in input Verifying content Verifying hsm state Verifying archive number is 2 PASS 10a (5s) == sanity-hsm test 10b: Archive of non dirty file must work without doing request ========================================================== 00:53:39 (1780808019) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_10b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' PASS 10b (6s) == sanity-hsm test 10c: Check forbidden archive ========== 00:53:45 (1780808025) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_10c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Cannot send HSM request (use of /mnt/lustre/d10c.sanity-hsm/f10c.sanity-hsm): Operation not permitted PASS 10c (3s) == sanity-hsm test 10d: Archive a file on the default archive id ========================================================== 00:53:48 (1780808028) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_10d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' PASS 10d (6s) == sanity-hsm test 11a: Import a file ==================== 00:53:54 (1780808034) lhsmtool_posix: 1780808035.613122 [22178]: action=1 src=d11a.sanity-hsm/f11a.sanity-hsm dst=/mnt/lustre/d11a.sanity-hsm/f11a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1780808035.619692 [22178]: importing '/mnt/lustre/d11a.sanity-hsm/f11a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_11a//d11a.sanity-hsm/f11a.sanity-hsm' lhsmtool_posix: 1780808035.652127 [22178]: imported '/mnt/lustre/d11a.sanity-hsm/f11a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_11a//0424/0x200000402:0x26:0x0'=='/tmp/arc1/sanity-hsm.test_11a//d11a.sanity-hsm/f11a.sanity-hsm' lhsmtool_posix: 1780808035.652205 [22178]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Verifying released state: Verifying imported size 158=158 Verifying released pattern: raid0,released Verifying new fid 0x200000402:0x26:0x0 in archive PASS 11a (4s) == sanity-hsm test 11b: Import a deleted file using its FID ========================================================== 00:53:58 (1780808038) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_11b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' lhsmtool_posix: 1780808041.561376 [22877]: action=1 src=0x200000403:0x11:0x0 dst=/mnt/lustre/d11b.sanity-hsm/f11b.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1780808041.573039 [22877]: Resolving [0x200000403:0x11:0x0] to /tmp/arc1/sanity-hsm.test_11b//0412/0x200000403:0x11:0x0 lhsmtool_posix: 1780808041.573594 [22877]: importing '/mnt/lustre/d11b.sanity-hsm/f11b.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_11b//0412/0x200000403:0x11:0x0' lhsmtool_posix: 1780808041.594018 [22877]: imported '/mnt/lustre/d11b.sanity-hsm/f11b.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_11b//042a/0x200000402:0x28:0x0'=='/tmp/arc1/sanity-hsm.test_11b//0412/0x200000403:0x11:0x0' lhsmtool_posix: 1780808041.594094 [22877]: process finished, errs: 0 major, 0 minor, rc=0 (Success) /mnt/lustre/d11b.sanity-hsm/f11b.sanity-hsm: OK PASS 11b (6s) == sanity-hsm test 11c: Import a file to a directory with a pool ========================================================== 00:54:04 (1780808044) Creating new pool test_11c oleg406-server: Pool lustre.test_11c created Adding targets to pool oleg406-server: OST lustre-OST0001_UUID added to pool lustre.test_11c lhsmtool_posix: 1780808052.184367 [23592]: action=1 src=d11c.sanity-hsm/f11c.sanity-hsm dst=/mnt/lustre/d11c.sanity-hsm/f11c.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1780808052.189808 [23592]: importing '/mnt/lustre/d11c.sanity-hsm/f11c.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_11c//d11c.sanity-hsm/f11c.sanity-hsm' lhsmtool_posix: 1780808052.231585 [23592]: imported '/mnt/lustre/d11c.sanity-hsm/f11c.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_11c//0429/0x200000402:0x2b:0x0'=='/tmp/arc1/sanity-hsm.test_11c//d11c.sanity-hsm/f11c.sanity-hsm' lhsmtool_posix: 1780808052.231646 [23592]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Destroy the created pools: test_11c lustre.test_11c oleg406-server: OST lustre-OST0001_UUID removed from pool lustre.test_11c oleg406-server: Pool lustre.test_11c destroyed PASS 11c (17s) == sanity-hsm test 12a: Restore an imported file explicitly ========================================================== 00:54:21 (1780808061) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1780808062.656390 [24328]: action=1 src=d12a.sanity-hsm/f12a.sanity-hsm dst=/mnt/lustre/d12a.sanity-hsm/f12a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1780808062.661196 [24328]: importing '/mnt/lustre/d12a.sanity-hsm/f12a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12a//d12a.sanity-hsm/f12a.sanity-hsm' lhsmtool_posix: 1780808062.703037 [24328]: imported '/mnt/lustre/d12a.sanity-hsm/f12a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12a//042c/0x200000402:0x2e:0x0'=='/tmp/arc1/sanity-hsm.test_12a//d12a.sanity-hsm/f12a.sanity-hsm' lhsmtool_posix: 1780808062.707352 [24328]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Verifying released state: Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Verifying file state: PASS 12a (6s) == sanity-hsm test 12b: Restore an imported file implicitly ========================================================== 00:54:27 (1780808067) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1780808068.910217 [25042]: action=1 src=d12b.sanity-hsm/f12b.sanity-hsm dst=/mnt/lustre/d12b.sanity-hsm/f12b.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1780808068.917803 [25042]: importing '/mnt/lustre/d12b.sanity-hsm/f12b.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12b//d12b.sanity-hsm/f12b.sanity-hsm' lhsmtool_posix: 1780808068.962400 [25042]: imported '/mnt/lustre/d12b.sanity-hsm/f12b.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12b//0432/0x200000402:0x30:0x0'=='/tmp/arc1/sanity-hsm.test_12b//d12b.sanity-hsm/f12b.sanity-hsm' lhsmtool_posix: 1780808068.962934 [25042]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Verifying released state: Verifying file state after restore: PASS 12b (6s) == sanity-hsm test 12c: Restore a file with stripe of 2 == 00:54:33 (1780808073) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.0758848 s, 69.1 MB/s Waiting 200s for 'SUCCEED' /mnt/lustre/d12c.sanity-hsm/f12c.sanity-hsm: OK PASS 12c (6s) == sanity-hsm test 12d: Restore of a non archived, non released file must work ========================================================== 00:54:39 (1780808079) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' PASS 12d (7s) == sanity-hsm test 12e: Check forbidden restore ========== 00:54:46 (1780808086) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12e/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' /mnt/lustre/d12e.sanity-hsm/f12e.sanity-hsm: (0x0000000b) exists dirty archived, archive_id:1 Cannot send HSM request (use of /mnt/lustre/d12e.sanity-hsm/f12e.sanity-hsm): Operation not permitted PASS 12e (6s) == sanity-hsm test 12f: Restore a released file explicitly ========================================================== 00:54:52 (1780808092) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12f/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Verifying file state: PASS 12f (7s) == sanity-hsm test 12g: Restore a released file implicitly ========================================================== 00:54:59 (1780808099) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12g/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' PASS 12g (6s) == sanity-hsm test 12h: Restore a released file implicitly from a second node ========================================================== 00:55:05 (1780808105) SKIP: sanity-hsm test_12h Need 2 or more clients, have 1 SKIP 12h (1s) == sanity-hsm test 12m: Archive/release/implicit restore ========================================================== 00:55:06 (1780808106) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12m/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' PASS 12m (6s) == sanity-hsm test 12n: Import/implicit restore/release == 00:55:12 (1780808112) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12n/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1780808113.187742 [29898]: action=1 src=d12n.sanity-hsm/f12n.sanity-hsm dst=/mnt/lustre/d12n.sanity-hsm/f12n.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1780808113.197325 [29898]: importing '/mnt/lustre/d12n.sanity-hsm/f12n.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12n//d12n.sanity-hsm/f12n.sanity-hsm' lhsmtool_posix: 1780808113.223768 [29898]: imported '/mnt/lustre/d12n.sanity-hsm/f12n.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12n//043d/0x200000402:0x3f:0x0'=='/tmp/arc1/sanity-hsm.test_12n//d12n.sanity-hsm/f12n.sanity-hsm' lhsmtool_posix: 1780808113.223860 [29898]: process finished, errs: 0 major, 0 minor, rc=0 (Success) PASS 12n (4s) == sanity-hsm test 12o: Layout-swap failure during Restore leaves file released ========================================================== 00:55:16 (1780808116) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12o/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' fail_loc=0x152 mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0001.hsm.policy=+NRA diff: /mnt/lustre/d12o.sanity-hsm/f12o.sanity-hsm: No data available mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA fail_loc=0 mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge Waiting 200s for 'SUCCEED' Updated after 5s: want 'SUCCEED' got 'SUCCEED' PASS 12o (27s) == sanity-hsm test 12p: implicit restore of a file on copytool mount point ========================================================== 00:55:43 (1780808143) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12p/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' PASS 12p (7s) == sanity-hsm test 12q: file attributes are refreshed after restore ========================================================== 00:55:50 (1780808150) Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre3 Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12q/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre3"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140731 s, 7.5 MB/s Waiting 200s for 'SUCCEED' Killing existing copytools on oleg406-client.virtnet with signal TERM copytools stopped in 0s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12q/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre3"' Stopping client oleg406-client.virtnet /mnt/lustre3 (opts:) PASS 12q (13s) == sanity-hsm test 12r: lseek restores released file ===== 00:56:03 (1780808163) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12r/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' PASS 12r (7s) == sanity-hsm test 12s: race between restore requests ==== 00:56:10 (1780808170) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12s/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' fail_loc=0x8000018b PASS 12s (10s) == sanity-hsm test 12t: Multiple parallel reads for a HSM imported file ========================================================== 00:56:20 (1780808180) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12t/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1780808182.446855 [34075]: action=1 src=0x200000403:0x2b:0x0 dst=/mnt/lustre/d12t.sanity-hsm/f12t.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1780808182.452738 [34075]: Resolving [0x200000403:0x2b:0x0] to /tmp/arc1/sanity-hsm.test_12t//0428/0x200000403:0x2b:0x0 lhsmtool_posix: 1780808182.452850 [34075]: importing '/mnt/lustre/d12t.sanity-hsm/f12t.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12t//0428/0x200000403:0x2b:0x0' lhsmtool_posix: 1780808182.483149 [34075]: imported '/mnt/lustre/d12t.sanity-hsm/f12t.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12t//044f/0x200000402:0x4d:0x0'=='/tmp/arc1/sanity-hsm.test_12t//0428/0x200000403:0x2b:0x0' lhsmtool_posix: 1780808182.483218 [34075]: process finished, errs: 0 major, 0 minor, rc=0 (Success) PASS 12t (5s) == sanity-hsm test 12u: Multiple reads on multiple HSM imported files in parallel ========================================================== 00:56:25 (1780808185) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12u/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1780808191.552281 [34943]: action=1 src=0x200000403:0x2d:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.0 mount_point=/mnt/lustre2 lhsmtool_posix: 1780808191.557807 [34943]: Resolving [0x200000403:0x2d:0x0] to /tmp/arc1/sanity-hsm.test_12u//042e/0x200000403:0x2d:0x0 lhsmtool_posix: 1780808191.557889 [34943]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.0' from '/tmp/arc1/sanity-hsm.test_12u//042e/0x200000403:0x2d:0x0' lhsmtool_posix: 1780808191.585852 [34943]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.0' from '/tmp/arc1/sanity-hsm.test_12u//044d/0x200000402:0x4f:0x0'=='/tmp/arc1/sanity-hsm.test_12u//042e/0x200000403:0x2d:0x0' lhsmtool_posix: 1780808191.585907 [34943]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1780808191.734523 [34972]: action=1 src=0x200000403:0x2e:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.1 mount_point=/mnt/lustre2 lhsmtool_posix: 1780808191.741598 [34972]: Resolving [0x200000403:0x2e:0x0] to /tmp/arc1/sanity-hsm.test_12u//042d/0x200000403:0x2e:0x0 lhsmtool_posix: 1780808191.741674 [34972]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.1' from '/tmp/arc1/sanity-hsm.test_12u//042d/0x200000403:0x2e:0x0' lhsmtool_posix: 1780808191.775562 [34972]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.1' from '/tmp/arc1/sanity-hsm.test_12u//0452/0x200000402:0x50:0x0'=='/tmp/arc1/sanity-hsm.test_12u//042d/0x200000403:0x2e:0x0' lhsmtool_posix: 1780808191.775874 [34972]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1780808191.943833 [35001]: action=1 src=0x200000403:0x2f:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.2 mount_point=/mnt/lustre2 lhsmtool_posix: 1780808191.950182 [35001]: Resolving [0x200000403:0x2f:0x0] to /tmp/arc1/sanity-hsm.test_12u//042c/0x200000403:0x2f:0x0 lhsmtool_posix: 1780808191.950276 [35001]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.2' from '/tmp/arc1/sanity-hsm.test_12u//042c/0x200000403:0x2f:0x0' lhsmtool_posix: 1780808191.994844 [35001]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.2' from '/tmp/arc1/sanity-hsm.test_12u//0453/0x200000402:0x51:0x0'=='/tmp/arc1/sanity-hsm.test_12u//042c/0x200000403:0x2f:0x0' lhsmtool_posix: 1780808191.994898 [35001]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1780808192.152017 [35030]: action=1 src=0x200000403:0x30:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.3 mount_point=/mnt/lustre2 lhsmtool_posix: 1780808192.155984 [35030]: Resolving [0x200000403:0x30:0x0] to /tmp/arc1/sanity-hsm.test_12u//0433/0x200000403:0x30:0x0 lhsmtool_posix: 1780808192.156060 [35030]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.3' from '/tmp/arc1/sanity-hsm.test_12u//0433/0x200000403:0x30:0x0' lhsmtool_posix: 1780808192.176625 [35030]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.3' from '/tmp/arc1/sanity-hsm.test_12u//0450/0x200000402:0x52:0x0'=='/tmp/arc1/sanity-hsm.test_12u//0433/0x200000403:0x30:0x0' lhsmtool_posix: 1780808192.176694 [35030]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1780808192.327928 [35059]: action=1 src=0x200000403:0x31:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.4 mount_point=/mnt/lustre2 lhsmtool_posix: 1780808192.334777 [35059]: Resolving [0x200000403:0x31:0x0] to /tmp/arc1/sanity-hsm.test_12u//0432/0x200000403:0x31:0x0 lhsmtool_posix: 1780808192.334860 [35059]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.4' from '/tmp/arc1/sanity-hsm.test_12u//0432/0x200000403:0x31:0x0' lhsmtool_posix: 1780808192.364052 [35059]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.4' from '/tmp/arc1/sanity-hsm.test_12u//0451/0x200000402:0x53:0x0'=='/tmp/arc1/sanity-hsm.test_12u//0432/0x200000403:0x31:0x0' lhsmtool_posix: 1780808192.364116 [35059]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1780808192.506411 [35088]: action=1 src=0x200000403:0x32:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.5 mount_point=/mnt/lustre2 lhsmtool_posix: 1780808192.512786 [35088]: Resolving [0x200000403:0x32:0x0] to /tmp/arc1/sanity-hsm.test_12u//0431/0x200000403:0x32:0x0 lhsmtool_posix: 1780808192.512944 [35088]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.5' from '/tmp/arc1/sanity-hsm.test_12u//0431/0x200000403:0x32:0x0' lhsmtool_posix: 1780808192.537821 [35088]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.5' from '/tmp/arc1/sanity-hsm.test_12u//0456/0x200000402:0x54:0x0'=='/tmp/arc1/sanity-hsm.test_12u//0431/0x200000403:0x32:0x0' lhsmtool_posix: 1780808192.537889 [35088]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1780808192.682054 [35117]: action=1 src=0x200000403:0x33:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.6 mount_point=/mnt/lustre2 lhsmtool_posix: 1780808192.686373 [35117]: Resolving [0x200000403:0x33:0x0] to /tmp/arc1/sanity-hsm.test_12u//0430/0x200000403:0x33:0x0 lhsmtool_posix: 1780808192.686483 [35117]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.6' from '/tmp/arc1/sanity-hsm.test_12u//0430/0x200000403:0x33:0x0' lhsmtool_posix: 1780808192.722486 [35117]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.6' from '/tmp/arc1/sanity-hsm.test_12u//0457/0x200000402:0x55:0x0'=='/tmp/arc1/sanity-hsm.test_12u//0430/0x200000403:0x33:0x0' lhsmtool_posix: 1780808192.722535 [35117]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1780808192.860610 [35146]: action=1 src=0x200000403:0x34:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.7 mount_point=/mnt/lustre2 lhsmtool_posix: 1780808192.866246 [35146]: Resolving [0x200000403:0x34:0x0] to /tmp/arc1/sanity-hsm.test_12u//0437/0x200000403:0x34:0x0 lhsmtool_posix: 1780808192.866336 [35146]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.7' from '/tmp/arc1/sanity-hsm.test_12u//0437/0x200000403:0x34:0x0' lhsmtool_posix: 1780808192.892058 [35146]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.7' from '/tmp/arc1/sanity-hsm.test_12u//0454/0x200000402:0x56:0x0'=='/tmp/arc1/sanity-hsm.test_12u//0437/0x200000403:0x34:0x0' lhsmtool_posix: 1780808192.892127 [35146]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1780808193.040827 [35175]: action=1 src=0x200000403:0x35:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.8 mount_point=/mnt/lustre2 lhsmtool_posix: 1780808193.044887 [35175]: Resolving [0x200000403:0x35:0x0] to /tmp/arc1/sanity-hsm.test_12u//0436/0x200000403:0x35:0x0 lhsmtool_posix: 1780808193.044970 [35175]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.8' from '/tmp/arc1/sanity-hsm.test_12u//0436/0x200000403:0x35:0x0' lhsmtool_posix: 1780808193.069199 [35175]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.8' from '/tmp/arc1/sanity-hsm.test_12u//0455/0x200000402:0x57:0x0'=='/tmp/arc1/sanity-hsm.test_12u//0436/0x200000403:0x35:0x0' lhsmtool_posix: 1780808193.069531 [35175]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1780808193.227078 [35204]: action=1 src=0x200000403:0x36:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.9 mount_point=/mnt/lustre2 lhsmtool_posix: 1780808193.233380 [35204]: Resolving [0x200000403:0x36:0x0] to /tmp/arc1/sanity-hsm.test_12u//0435/0x200000403:0x36:0x0 lhsmtool_posix: 1780808193.233482 [35204]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.9' from '/tmp/arc1/sanity-hsm.test_12u//0435/0x200000403:0x36:0x0' lhsmtool_posix: 1780808193.268900 [35204]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.9' from '/tmp/arc1/sanity-hsm.test_12u//045a/0x200000402:0x58:0x0'=='/tmp/arc1/sanity-hsm.test_12u//0435/0x200000403:0x36:0x0' lhsmtool_posix: 1780808193.269234 [35204]: process finished, errs: 0 major, 0 minor, rc=0 (Success) PASS 12u (12s) == sanity-hsm test 13: Recursively import and restore a directory ========================================================== 00:56:37 (1780808197) lhsmtool_posix: 1780808200.877637 [36954]: action=1 src=subdir dst=/mnt/lustre/d13.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1780808200.995153 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.10' lhsmtool_posix: 1780808201.055661 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.10' from '/tmp/arc1/sanity-hsm.test_13//0458/0x200000402:0x5a:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.10' lhsmtool_posix: 1780808201.056379 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.9' lhsmtool_posix: 1780808201.081119 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.9' from '/tmp/arc1/sanity-hsm.test_13//0459/0x200000402:0x5b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.9' lhsmtool_posix: 1780808201.081379 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.8' lhsmtool_posix: 1780808201.112281 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.8' from '/tmp/arc1/sanity-hsm.test_13//045e/0x200000402:0x5c:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.8' lhsmtool_posix: 1780808201.112517 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.7' lhsmtool_posix: 1780808201.134386 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.7' from '/tmp/arc1/sanity-hsm.test_13//045f/0x200000402:0x5d:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.7' lhsmtool_posix: 1780808201.134620 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.6' lhsmtool_posix: 1780808201.155706 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.6' from '/tmp/arc1/sanity-hsm.test_13//045c/0x200000402:0x5e:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.6' lhsmtool_posix: 1780808201.155859 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.5' lhsmtool_posix: 1780808201.180339 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.5' from '/tmp/arc1/sanity-hsm.test_13//045d/0x200000402:0x5f:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.5' lhsmtool_posix: 1780808201.180585 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.4' lhsmtool_posix: 1780808201.199705 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.4' from '/tmp/arc1/sanity-hsm.test_13//0462/0x200000402:0x60:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.4' lhsmtool_posix: 1780808201.199902 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.3' lhsmtool_posix: 1780808201.217655 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.3' from '/tmp/arc1/sanity-hsm.test_13//0463/0x200000402:0x61:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.3' lhsmtool_posix: 1780808201.217851 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.2' lhsmtool_posix: 1780808201.241318 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.2' from '/tmp/arc1/sanity-hsm.test_13//0460/0x200000402:0x62:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.2' lhsmtool_posix: 1780808201.241508 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.1' lhsmtool_posix: 1780808201.264772 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.1' from '/tmp/arc1/sanity-hsm.test_13//0461/0x200000402:0x63:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.1' lhsmtool_posix: 1780808201.269206 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.10' lhsmtool_posix: 1780808201.297156 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.10' from '/tmp/arc1/sanity-hsm.test_13//040b/0x240000402:0x9:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.10' lhsmtool_posix: 1780808201.297402 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.9' lhsmtool_posix: 1780808201.315582 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.9' from '/tmp/arc1/sanity-hsm.test_13//0408/0x240000402:0xa:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.9' lhsmtool_posix: 1780808201.315800 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.8' lhsmtool_posix: 1780808201.334253 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.8' from '/tmp/arc1/sanity-hsm.test_13//0409/0x240000402:0xb:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.8' lhsmtool_posix: 1780808201.334476 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.7' lhsmtool_posix: 1780808201.353347 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.7' from '/tmp/arc1/sanity-hsm.test_13//040e/0x240000402:0xc:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.7' lhsmtool_posix: 1780808201.353591 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.6' lhsmtool_posix: 1780808201.374552 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.6' from '/tmp/arc1/sanity-hsm.test_13//040f/0x240000402:0xd:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.6' lhsmtool_posix: 1780808201.374797 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.5' lhsmtool_posix: 1780808201.398035 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.5' from '/tmp/arc1/sanity-hsm.test_13//040c/0x240000402:0xe:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.5' lhsmtool_posix: 1780808201.398278 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.4' lhsmtool_posix: 1780808201.420326 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.4' from '/tmp/arc1/sanity-hsm.test_13//040d/0x240000402:0xf:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.4' lhsmtool_posix: 1780808201.421470 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.3' lhsmtool_posix: 1780808201.444533 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.3' from '/tmp/arc1/sanity-hsm.test_13//0412/0x240000402:0x10:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.3' lhsmtool_posix: 1780808201.444763 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.2' lhsmtool_posix: 1780808201.466086 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.2' from '/tmp/arc1/sanity-hsm.test_13//0413/0x240000402:0x11:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.2' lhsmtool_posix: 1780808201.466327 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.1' lhsmtool_posix: 1780808201.488353 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.1' from '/tmp/arc1/sanity-hsm.test_13//0410/0x240000402:0x12:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.1' lhsmtool_posix: 1780808201.500331 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.10' lhsmtool_posix: 1780808201.581362 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.10' from '/tmp/arc1/sanity-hsm.test_13//0467/0x200000402:0x65:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.10' lhsmtool_posix: 1780808201.581814 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.9' lhsmtool_posix: 1780808201.598889 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.9' from '/tmp/arc1/sanity-hsm.test_13//0464/0x200000402:0x66:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.9' lhsmtool_posix: 1780808201.599131 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.8' lhsmtool_posix: 1780808201.620884 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.8' from '/tmp/arc1/sanity-hsm.test_13//0465/0x200000402:0x67:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.8' lhsmtool_posix: 1780808201.621138 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.7' lhsmtool_posix: 1780808201.645495 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.7' from '/tmp/arc1/sanity-hsm.test_13//046a/0x200000402:0x68:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.7' lhsmtool_posix: 1780808201.645724 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.6' lhsmtool_posix: 1780808201.664230 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.6' from '/tmp/arc1/sanity-hsm.test_13//046b/0x200000402:0x69:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.6' lhsmtool_posix: 1780808201.664580 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.5' lhsmtool_posix: 1780808201.681468 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.5' from '/tmp/arc1/sanity-hsm.test_13//0468/0x200000402:0x6a:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.5' lhsmtool_posix: 1780808201.681652 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.4' lhsmtool_posix: 1780808201.700478 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.4' from '/tmp/arc1/sanity-hsm.test_13//0469/0x200000402:0x6b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.4' lhsmtool_posix: 1780808201.700652 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.3' lhsmtool_posix: 1780808201.722053 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.3' from '/tmp/arc1/sanity-hsm.test_13//046e/0x200000402:0x6c:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.3' lhsmtool_posix: 1780808201.722782 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.2' lhsmtool_posix: 1780808201.752350 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.2' from '/tmp/arc1/sanity-hsm.test_13//046f/0x200000402:0x6d:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.2' lhsmtool_posix: 1780808201.752689 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.1' lhsmtool_posix: 1780808201.794610 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.1' from '/tmp/arc1/sanity-hsm.test_13//046c/0x200000402:0x6e:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.1' lhsmtool_posix: 1780808201.799772 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.10' lhsmtool_posix: 1780808201.833312 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.10' from '/tmp/arc1/sanity-hsm.test_13//0416/0x240000402:0x14:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.10' lhsmtool_posix: 1780808201.833514 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.9' lhsmtool_posix: 1780808201.872394 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.9' from '/tmp/arc1/sanity-hsm.test_13//0417/0x240000402:0x15:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.9' lhsmtool_posix: 1780808201.873048 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.8' lhsmtool_posix: 1780808201.906685 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.8' from '/tmp/arc1/sanity-hsm.test_13//0414/0x240000402:0x16:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.8' lhsmtool_posix: 1780808201.906845 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.7' lhsmtool_posix: 1780808201.935335 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.7' from '/tmp/arc1/sanity-hsm.test_13//0415/0x240000402:0x17:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.7' lhsmtool_posix: 1780808201.935562 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.6' lhsmtool_posix: 1780808201.957017 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.6' from '/tmp/arc1/sanity-hsm.test_13//041a/0x240000402:0x18:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.6' lhsmtool_posix: 1780808201.957255 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.5' lhsmtool_posix: 1780808201.984717 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.5' from '/tmp/arc1/sanity-hsm.test_13//041b/0x240000402:0x19:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.5' lhsmtool_posix: 1780808201.985543 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.4' lhsmtool_posix: 1780808202.005603 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.4' from '/tmp/arc1/sanity-hsm.test_13//0418/0x240000402:0x1a:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.4' lhsmtool_posix: 1780808202.005788 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.3' lhsmtool_posix: 1780808202.026904 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.3' from '/tmp/arc1/sanity-hsm.test_13//0419/0x240000402:0x1b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.3' lhsmtool_posix: 1780808202.027101 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.2' lhsmtool_posix: 1780808202.049824 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.2' from '/tmp/arc1/sanity-hsm.test_13//041e/0x240000402:0x1c:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.2' lhsmtool_posix: 1780808202.050119 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.1' lhsmtool_posix: 1780808202.071217 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.1' from '/tmp/arc1/sanity-hsm.test_13//041f/0x240000402:0x1d:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.1' lhsmtool_posix: 1780808202.074205 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.10' lhsmtool_posix: 1780808202.094713 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.10' from '/tmp/arc1/sanity-hsm.test_13//041d/0x240000402:0x1f:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.10' lhsmtool_posix: 1780808202.094962 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.9' lhsmtool_posix: 1780808202.116887 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.9' from '/tmp/arc1/sanity-hsm.test_13//0422/0x240000402:0x20:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.9' lhsmtool_posix: 1780808202.117106 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.8' lhsmtool_posix: 1780808202.137768 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.8' from '/tmp/arc1/sanity-hsm.test_13//0423/0x240000402:0x21:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.8' lhsmtool_posix: 1780808202.137977 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.7' lhsmtool_posix: 1780808202.157847 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.7' from '/tmp/arc1/sanity-hsm.test_13//0420/0x240000402:0x22:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.7' lhsmtool_posix: 1780808202.158069 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.6' lhsmtool_posix: 1780808202.179806 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.6' from '/tmp/arc1/sanity-hsm.test_13//0421/0x240000402:0x23:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.6' lhsmtool_posix: 1780808202.180355 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.5' lhsmtool_posix: 1780808202.203293 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.5' from '/tmp/arc1/sanity-hsm.test_13//0426/0x240000402:0x24:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.5' lhsmtool_posix: 1780808202.203612 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.4' lhsmtool_posix: 1780808202.225346 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.4' from '/tmp/arc1/sanity-hsm.test_13//0427/0x240000402:0x25:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.4' lhsmtool_posix: 1780808202.225617 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.3' lhsmtool_posix: 1780808202.248221 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.3' from '/tmp/arc1/sanity-hsm.test_13//0424/0x240000402:0x26:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.3' lhsmtool_posix: 1780808202.248408 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.2' lhsmtool_posix: 1780808202.270782 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.2' from '/tmp/arc1/sanity-hsm.test_13//0425/0x240000402:0x27:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.2' lhsmtool_posix: 1780808202.270979 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.1' lhsmtool_posix: 1780808202.292662 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.1' from '/tmp/arc1/sanity-hsm.test_13//042a/0x240000402:0x28:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.1' lhsmtool_posix: 1780808202.302567 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.10' lhsmtool_posix: 1780808202.359935 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.10' from '/tmp/arc1/sanity-hsm.test_13//0472/0x200000402:0x70:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.10' lhsmtool_posix: 1780808202.360157 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.9' lhsmtool_posix: 1780808202.386271 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.9' from '/tmp/arc1/sanity-hsm.test_13//0473/0x200000402:0x71:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.9' lhsmtool_posix: 1780808202.386649 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.8' lhsmtool_posix: 1780808202.410483 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.8' from '/tmp/arc1/sanity-hsm.test_13//0470/0x200000402:0x72:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.8' lhsmtool_posix: 1780808202.410711 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.7' lhsmtool_posix: 1780808202.435502 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.7' from '/tmp/arc1/sanity-hsm.test_13//0471/0x200000402:0x73:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.7' lhsmtool_posix: 1780808202.435745 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.6' lhsmtool_posix: 1780808202.461249 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.6' from '/tmp/arc1/sanity-hsm.test_13//0476/0x200000402:0x74:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.6' lhsmtool_posix: 1780808202.461949 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.5' lhsmtool_posix: 1780808202.487674 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.5' from '/tmp/arc1/sanity-hsm.test_13//0477/0x200000402:0x75:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.5' lhsmtool_posix: 1780808202.488012 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.4' lhsmtool_posix: 1780808202.516746 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.4' from '/tmp/arc1/sanity-hsm.test_13//0474/0x200000402:0x76:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.4' lhsmtool_posix: 1780808202.517013 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.3' lhsmtool_posix: 1780808202.541640 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.3' from '/tmp/arc1/sanity-hsm.test_13//0475/0x200000402:0x77:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.3' lhsmtool_posix: 1780808202.543813 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.2' lhsmtool_posix: 1780808202.560837 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.2' from '/tmp/arc1/sanity-hsm.test_13//047a/0x200000402:0x78:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.2' lhsmtool_posix: 1780808202.561069 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.1' lhsmtool_posix: 1780808202.585202 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.1' from '/tmp/arc1/sanity-hsm.test_13//047b/0x200000402:0x79:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.1' lhsmtool_posix: 1780808202.596884 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.10' lhsmtool_posix: 1780808202.644828 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.10' from '/tmp/arc1/sanity-hsm.test_13//0479/0x200000402:0x7b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.10' lhsmtool_posix: 1780808202.645014 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.9' lhsmtool_posix: 1780808202.666532 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.9' from '/tmp/arc1/sanity-hsm.test_13//047e/0x200000402:0x7c:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.9' lhsmtool_posix: 1780808202.666752 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.8' lhsmtool_posix: 1780808202.686789 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.8' from '/tmp/arc1/sanity-hsm.test_13//047f/0x200000402:0x7d:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.8' lhsmtool_posix: 1780808202.687611 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.7' lhsmtool_posix: 1780808202.711504 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.7' from '/tmp/arc1/sanity-hsm.test_13//047c/0x200000402:0x7e:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.7' lhsmtool_posix: 1780808202.711764 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.6' lhsmtool_posix: 1780808202.731665 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.6' from '/tmp/arc1/sanity-hsm.test_13//047d/0x200000402:0x7f:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.6' lhsmtool_posix: 1780808202.731924 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.5' lhsmtool_posix: 1780808202.754297 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.5' from '/tmp/arc1/sanity-hsm.test_13//0482/0x200000402:0x80:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.5' lhsmtool_posix: 1780808202.754618 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.4' lhsmtool_posix: 1780808202.781701 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.4' from '/tmp/arc1/sanity-hsm.test_13//0483/0x200000402:0x81:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.4' lhsmtool_posix: 1780808202.782165 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.3' lhsmtool_posix: 1780808202.804157 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.3' from '/tmp/arc1/sanity-hsm.test_13//0480/0x200000402:0x82:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.3' lhsmtool_posix: 1780808202.804360 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.2' lhsmtool_posix: 1780808202.827115 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.2' from '/tmp/arc1/sanity-hsm.test_13//0481/0x200000402:0x83:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.2' lhsmtool_posix: 1780808202.827327 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.1' lhsmtool_posix: 1780808202.854803 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.1' from '/tmp/arc1/sanity-hsm.test_13//0486/0x200000402:0x84:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.1' lhsmtool_posix: 1780808202.864911 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.10' lhsmtool_posix: 1780808202.919175 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.10' from '/tmp/arc1/sanity-hsm.test_13//0484/0x200000402:0x86:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.10' lhsmtool_posix: 1780808202.919369 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.9' lhsmtool_posix: 1780808202.934530 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.9' from '/tmp/arc1/sanity-hsm.test_13//0485/0x200000402:0x87:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.9' lhsmtool_posix: 1780808202.934792 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.8' lhsmtool_posix: 1780808202.950137 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.8' from '/tmp/arc1/sanity-hsm.test_13//048a/0x200000402:0x88:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.8' lhsmtool_posix: 1780808202.950394 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.7' lhsmtool_posix: 1780808202.971659 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.7' from '/tmp/arc1/sanity-hsm.test_13//048b/0x200000402:0x89:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.7' lhsmtool_posix: 1780808202.971863 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.6' lhsmtool_posix: 1780808202.992037 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.6' from '/tmp/arc1/sanity-hsm.test_13//0488/0x200000402:0x8a:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.6' lhsmtool_posix: 1780808202.992339 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.5' lhsmtool_posix: 1780808203.009818 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.5' from '/tmp/arc1/sanity-hsm.test_13//0489/0x200000402:0x8b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.5' lhsmtool_posix: 1780808203.010024 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.4' lhsmtool_posix: 1780808203.026323 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.4' from '/tmp/arc1/sanity-hsm.test_13//048e/0x200000402:0x8c:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.4' lhsmtool_posix: 1780808203.026556 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.3' lhsmtool_posix: 1780808203.042231 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.3' from '/tmp/arc1/sanity-hsm.test_13//048f/0x200000402:0x8d:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.3' lhsmtool_posix: 1780808203.042471 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.2' lhsmtool_posix: 1780808203.055741 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.2' from '/tmp/arc1/sanity-hsm.test_13//048c/0x200000402:0x8e:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.2' lhsmtool_posix: 1780808203.055926 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.1' lhsmtool_posix: 1780808203.072272 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.1' from '/tmp/arc1/sanity-hsm.test_13//048d/0x200000402:0x8f:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.1' lhsmtool_posix: 1780808203.075708 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.10' lhsmtool_posix: 1780808203.099767 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.10' from '/tmp/arc1/sanity-hsm.test_13//0428/0x240000402:0x2a:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.10' lhsmtool_posix: 1780808203.100005 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.9' lhsmtool_posix: 1780808203.118950 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.9' from '/tmp/arc1/sanity-hsm.test_13//0429/0x240000402:0x2b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.9' lhsmtool_posix: 1780808203.119165 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.8' lhsmtool_posix: 1780808203.141628 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.8' from '/tmp/arc1/sanity-hsm.test_13//042e/0x240000402:0x2c:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.8' lhsmtool_posix: 1780808203.141818 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.7' lhsmtool_posix: 1780808203.163380 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.7' from '/tmp/arc1/sanity-hsm.test_13//042f/0x240000402:0x2d:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.7' lhsmtool_posix: 1780808203.163697 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.6' lhsmtool_posix: 1780808203.187948 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.6' from '/tmp/arc1/sanity-hsm.test_13//042c/0x240000402:0x2e:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.6' lhsmtool_posix: 1780808203.188227 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.5' lhsmtool_posix: 1780808203.213309 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.5' from '/tmp/arc1/sanity-hsm.test_13//042d/0x240000402:0x2f:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.5' lhsmtool_posix: 1780808203.213592 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.4' lhsmtool_posix: 1780808203.234245 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.4' from '/tmp/arc1/sanity-hsm.test_13//0432/0x240000402:0x30:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.4' lhsmtool_posix: 1780808203.234412 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.3' lhsmtool_posix: 1780808203.249342 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.3' from '/tmp/arc1/sanity-hsm.test_13//0433/0x240000402:0x31:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.3' lhsmtool_posix: 1780808203.249605 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.2' lhsmtool_posix: 1780808203.264260 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.2' from '/tmp/arc1/sanity-hsm.test_13//0430/0x240000402:0x32:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.2' lhsmtool_posix: 1780808203.264652 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.1' lhsmtool_posix: 1780808203.277701 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.1' from '/tmp/arc1/sanity-hsm.test_13//0431/0x240000402:0x33:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.1' lhsmtool_posix: 1780808203.283248 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.10' lhsmtool_posix: 1780808203.314945 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.10' from '/tmp/arc1/sanity-hsm.test_13//0493/0x200000402:0x91:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.10' lhsmtool_posix: 1780808203.315141 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.9' lhsmtool_posix: 1780808203.328840 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.9' from '/tmp/arc1/sanity-hsm.test_13//0490/0x200000402:0x92:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.9' lhsmtool_posix: 1780808203.329033 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.8' lhsmtool_posix: 1780808203.342319 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.8' from '/tmp/arc1/sanity-hsm.test_13//0491/0x200000402:0x93:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.8' lhsmtool_posix: 1780808203.342516 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.7' lhsmtool_posix: 1780808203.357895 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.7' from '/tmp/arc1/sanity-hsm.test_13//0496/0x200000402:0x94:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.7' lhsmtool_posix: 1780808203.358132 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.6' lhsmtool_posix: 1780808203.373869 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.6' from '/tmp/arc1/sanity-hsm.test_13//0497/0x200000402:0x95:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.6' lhsmtool_posix: 1780808203.374058 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.5' lhsmtool_posix: 1780808203.387610 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.5' from '/tmp/arc1/sanity-hsm.test_13//0494/0x200000402:0x96:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.5' lhsmtool_posix: 1780808203.387747 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.4' lhsmtool_posix: 1780808203.400047 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.4' from '/tmp/arc1/sanity-hsm.test_13//0495/0x200000402:0x97:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.4' lhsmtool_posix: 1780808203.400278 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.3' lhsmtool_posix: 1780808203.416234 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.3' from '/tmp/arc1/sanity-hsm.test_13//049a/0x200000402:0x98:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.3' lhsmtool_posix: 1780808203.416472 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.2' lhsmtool_posix: 1780808203.431776 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.2' from '/tmp/arc1/sanity-hsm.test_13//049b/0x200000402:0x99:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.2' lhsmtool_posix: 1780808203.431944 [36954]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.1' lhsmtool_posix: 1780808203.446242 [36954]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.1' from '/tmp/arc1/sanity-hsm.test_13//0498/0x200000402:0x9a:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.1' lhsmtool_posix: 1780808203.446332 [36954]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_13/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' PASS 13 (11s) == sanity-hsm test 14: Rebind archived file to a new fid ========================================================== 00:56:48 (1780808208) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_14/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736645 s, 14.2 MB/s Waiting 200s for 'SUCCEED' 0+0 records in 0+0 records out 0 bytes copied, 0.00214568 s, 0.0 kB/s rebind 0x200000403:0x7d:0x0 to 0x200000403:0x7e:0x0 lhsmtool_posix: 1780808211.444364 [37908]: action=2 src=0x200000403:0x7d:0x0 dst=0x200000403:0x7e:0x0 mount_point=/mnt/lustre2 lhsmtool_posix: 1780808211.445314 [37908]: rebind [0x200000403:0x7d:0x0] to [0x200000403:0x7e:0x0] lhsmtool_posix: 1780808211.445470 [37908]: process finished, errs: 0 major, 0 minor, rc=0 (Success) PASS 14 (6s) == sanity-hsm test 15: Rebind a list of files ============ 00:56:54 (1780808214) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_15/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811231 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866261 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727081 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069193 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814698 s, 12.9 MB/s pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 0+0 records in 0+0 records out 0 bytes copied, 0.00752722 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00213369 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.0041953 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00427045 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00347054 s, 0.0 kB/s rebind list of files lhsmtool_posix: 1780808217.468588 [38689]: action=2 src=/tmp/tmp.8369 dst=(null) mount_point=/mnt/lustre2 lhsmtool_posix: 1780808217.469850 [38689]: rebind [0x200000403:0x80:0x0] to [0x200000403:0x85:0x0] lhsmtool_posix: 1780808217.469988 [38689]: rebind [0x200000403:0x81:0x0] to [0x200000403:0x86:0x0] lhsmtool_posix: 1780808217.470039 [38689]: rebind [0x200000403:0x82:0x0] to [0x200000403:0x87:0x0] lhsmtool_posix: 1780808217.470084 [38689]: rebind [0x200000403:0x83:0x0] to [0x200000403:0x88:0x0] lhsmtool_posix: 1780808217.470132 [38689]: rebind [0x200000403:0x84:0x0] to [0x200000403:0x89:0x0] lhsmtool_posix: 1780808217.470185 [38689]: 5 lines read from '/tmp/tmp.8369', 5 rebind successful lhsmtool_posix: 1780808217.470202 [38689]: process finished, errs: 0 major, 0 minor, rc=0 (Success) PASS 15 (10s) == sanity-hsm test 16: Test CT bandwith control option === 00:57:04 (1780808224) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_16/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.0346346 s, 606 MB/s Waiting 200s for 'SUCCEED' Changed after 1s: from 'WAITING' to 'STARTED' Waiting 190s for 'SUCCEED' Waiting 180s for 'SUCCEED' Updated after 21s: want 'SUCCEED' got 'SUCCEED' PASS 16 (26s) == sanity-hsm test 20: Release is not permitted ========== 00:57:30 (1780808250) 0+0 records in 0+0 records out 0 bytes copied, 0.00253723 s, 0.0 kB/s 0x240000403:0x2d:0x0 Cannot send HSM request (use of /mnt/lustre/d20.sanity-hsm/f20.sanity-hsm): Operation not permitted Cannot send HSM request (use of /mnt/lustre/d20.sanity-hsm/f20.sanity-hsm): Operation not permitted Cannot send HSM request (use of /mnt/lustre/d20.sanity-hsm/f20.sanity-hsm): Operation not permitted Cannot send HSM request (use of /mnt/lustre/d20.sanity-hsm/f20.sanity-hsm): Operation not permitted PASS 20 (3s) == sanity-hsm test 21: Simple release tests ============== 00:57:33 (1780808253) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_21/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061142 s, 17.1 MB/s 1+0 records in 1+0 records out 512 bytes copied, 0.0254158 s, 20.1 kB/s Waiting 200s for 'SUCCEED' PASS 21 (5s) == sanity-hsm test 22: Could not swap a release file ===== 00:57:38 (1780808258) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_22/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542334 s, 19.3 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555757 s, 18.9 MB/s 0x200000403:0x92:0x0 lfs swap_layouts: error: cannot swap layout between '/mnt/lustre/d22.sanity-hsm/test_swap' and '/mnt/lustre/d22.sanity-hsm/test_release': Operation not supported (95) PASS 22 (4s) == sanity-hsm test 23: Release does not change a/mtime (utime) ========================================================== 00:57:42 (1780808262) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_23/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557012 s, 18.8 MB/s Waiting 200s for 'SUCCEED' PASS 23 (4s) == sanity-hsm test 24a: Archive, release, and restore does not change a/mtime (i/o) ========================================================== 00:57:46 (1780808266) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591102 s, 17.7 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Killing existing copytools on oleg406-client.virtnet with signal TERM copytools stopped in 0s 192.168.204.106@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg406-client.virtnet /mnt/lustre (opts:) Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre PASS 24a (7s) == sanity-hsm test 24b: root can archive, release, and restore user files ========================================================== 00:57:53 (1780808273) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901207 s, 11.6 MB/s Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d24b.sanity-hsm/f24b.sanity-hsm] /mnt/lustre/d24b.sanity-hsm/f24b.sanity-hsm: (0x00000009) exists archived, archive_id:1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [md5sum] [/mnt/lustre/d24b.sanity-hsm/f24b.sanity-hsm] PASS 24b (7s) == sanity-hsm test 24c: check that user,group,other request masks work ========================================================== 00:58:00 (1780808280) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562722 s, 18.6 MB/s 0x200000403:0x9b:0x0 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_archive] [/mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm] Cannot send HSM request (use of /mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm): Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_archive] [/mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576956 s, 18.2 MB/s 0x200000403:0x9c:0x0 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_archive] [/mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm] Cannot send HSM request (use of /mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm): Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_archive] [/mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055436 s, 18.9 MB/s 0x200000403:0x9d:0x0 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_archive] [/mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm] Cannot send HSM request (use of /mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm): Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_archive] [/mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm] PASS 24c (7s) == sanity-hsm test 24d: check that read-only mounts are respected ========================================================== 00:58:07 (1780808287) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583232 s, 18.0 MB/s 0x200000403:0x9e:0x0 /mnt/lustre/d24d.sanity-hsm/f24d.sanity-hsm lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 85 0x55 0x280000401 Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre3 Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre3"' Cannot send HSM request (use of /mnt/lustre2/d24d.sanity-hsm/f24d.sanity-hsm): Read-only file system Waiting 200s for 'SUCCEED' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 1.08899 s, 963 kB/s Cannot send HSM request (use of /mnt/lustre2/d24d.sanity-hsm/f24d.sanity-hsm): Read-only file system Stopping client oleg406-client.virtnet /mnt/lustre3 (opts:) PASS 24d (7s) == sanity-hsm test 24e: tar succeeds on HSM released files ========================================================== 00:58:14 (1780808294) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24e/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690632 s, 15.2 MB/s /mnt/lustre/d24e.sanity-hsm/f24e.sanity-hsm: (0x0000000d) released exists archived, archive_id:1 tar: Removing leading `/' from member names PASS 24e (4s) == sanity-hsm test 24f: root can archive, release, and restore tar files ========================================================== 00:58:18 (1780808298) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24f/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 54fb6627dbaa37721048e4549db3224d /mnt/lustre/d24f.sanity-hsm/f24f.sanity-hsm Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' f24f.sanity-hsm f24f.sanity-hsm Sum0 = 54fb6627dbaa37721048e4549db3224d /mnt/lustre/d24f.sanity-hsm/f24f.sanity-hsm, sum1 = 54fb6627dbaa37721048e4549db3224d /mnt/lustre/d24f.sanity-hsm/f24f.sanity-hsm PASS 24f (4s) == sanity-hsm test 24g: write by non-owner still sets dirty ========================================================== 00:58:22 (1780808302) RUNAS = 'runas -u 500 -g 500' Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24g/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Please listen carefully as our options have changed. Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' running as uid/gid/euid/egid 500/500/500/500, groups: 500 [tee] [/mnt/lustre/d24g.sanity-hsm/f24g.sanity-hsm] To be electrocuted by your telephone, press #. PASS 24g (4s) == sanity-hsm test 25a: Restore lost file (HS_LOST flag) from import ========================================================== 00:58:26 (1780808306) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_25a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1780808307.000921 [47765]: action=1 src=d25a.sanity-hsm/f25a.sanity-hsm dst=/mnt/lustre/d25a.sanity-hsm/f25a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1780808307.003982 [47765]: importing '/mnt/lustre/d25a.sanity-hsm/f25a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_25a//d25a.sanity-hsm/f25a.sanity-hsm' lhsmtool_posix: 1780808307.018867 [47765]: imported '/mnt/lustre/d25a.sanity-hsm/f25a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_25a//0416/0x200000405:0x13:0x0'=='/tmp/arc1/sanity-hsm.test_25a//d25a.sanity-hsm/f25a.sanity-hsm' lhsmtool_posix: 1780808307.018923 [47765]: process finished, errs: 0 major, 0 minor, rc=0 (Success) md5sum: /mnt/lustre/d25a.sanity-hsm/f25a.sanity-hsm: No data available PASS 25a (2s) == sanity-hsm test 25b: Restore lost file (HS_LOST flag) after release ========================================================== 00:58:28 (1780808308) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_25b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' md5sum: /mnt/lustre/d25b.sanity-hsm/f25b.sanity-hsm: No data available PASS 25b (4s) == sanity-hsm test 26A: Remove the archive of a valid file ========================================================== 00:58:33 (1780808313) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26A/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes copied, 0.0018579 s, 0.0 kB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Waiting 200s for 'SUCCEED' PASS 26A (6s) == sanity-hsm test 26a: Remove Archive On Last Unlink (RAoLU) policy ========================================================== 00:58:39 (1780808319) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' Updated after 5s: want 'SUCCEED' got 'SUCCEED' PASS 26a (16s) == sanity-hsm test 26b: RAoLU policy when CDT off ======== 00:58:55 (1780808335) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled Killing existing copytools on oleg406-client.virtnet with signal TERM copytools stopped in 0s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' PASS 26b (11s) == sanity-hsm test 26c: RAoLU effective when file closed ========================================================== 00:59:06 (1780808346) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' multiop /mnt/lustre/d26c.sanity-hsm/f26c.sanity-hsm vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8369 Waiting 200s for 'SUCCEED' Updated after 8s: want 'SUCCEED' got 'SUCCEED' Waiting 200s for 'SUCCEED' Updated after 10s: want 'SUCCEED' got 'SUCCEED' PASS 26c (30s) == sanity-hsm test 26d: RAoLU when Client eviction ======= 00:59:36 (1780808376) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635404 s, 16.5 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' multiop /mnt/lustre/d26d.sanity-hsm/f26d.sanity-hsm vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8369 pdsh@oleg406-client: oleg406-client: ssh exited with exit code 5 PASS 26d (13s) == sanity-hsm test 26e: RAoLU with a non-started coordinator ========================================================== 00:59:49 (1780808389) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26e/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767977 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800557 s, 13.1 MB/s Waiting 200s for 'SUCCEED' Killing existing copytools on oleg406-client.virtnet with signal TERM copytools stopped in 0s Failing mds1 on oleg406-server Stopping /mnt/lustre-mds1 (opts:) on oleg406-server 01:00:15 (1780808415) shut down facet: mds1 facet_host: oleg406-server facet_failover_host: oleg406-server Failover mds1 to oleg406-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 01:00:29 (1780808429) targets are mounted 01:00:29 (1780808429) facet_failover done oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Check cookie from RAoLU request (last: 1780808091, remove: 1780808092) mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26e/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' PASS 26e (75s) == sanity-hsm test 27a: Remove the archive of an imported file (Operation not permitted) ========================================================== 01:01:04 (1780808464) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_27a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 39+0 records in 39+0 records out 40894464 bytes (41 MB, 39 MiB) copied, 0.43697 s, 93.6 MB/s lhsmtool_posix: 1780808466.267486 [54989]: action=1 src=d27a.sanity-hsm/f27a.sanity-hsm dst=/mnt/lustre/d27a.sanity-hsm/f27a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1780808466.269909 [54989]: importing '/mnt/lustre/d27a.sanity-hsm/f27a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_27a//d27a.sanity-hsm/f27a.sanity-hsm' lhsmtool_posix: 1780808466.284172 [54989]: imported '/mnt/lustre/d27a.sanity-hsm/f27a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_27a//0404/0x200000407:0x3:0x0'=='/tmp/arc1/sanity-hsm.test_27a//d27a.sanity-hsm/f27a.sanity-hsm' lhsmtool_posix: 1780808466.284238 [54989]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Cannot send HSM request (use of /mnt/lustre/d27a.sanity-hsm/f27a.sanity-hsm): Operation not permitted PASS 27a (4s) == sanity-hsm test 27b: Remove the archive of a relased file (Operation not permitted) ========================================================== 01:01:08 (1780808468) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_27b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes copied, 0.00189521 s, 0.0 kB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Cannot send HSM request (use of /mnt/lustre/d27b.sanity-hsm/f27b.sanity-hsm): Operation not permitted PASS 27b (4s) == sanity-hsm test 28: Concurrent archive/file remove ==== 01:01:12 (1780808472) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_28/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes copied, 0.00152133 s, 0.0 kB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 28 (6s) == sanity-hsm test 29a: Tests --mntpath and --archive options ========================================================== 01:01:18 (1780808478) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_29a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 7 "/mnt/lustre"' Cannot send HSM request (use of /mnt/lustre): Invalid argument hsm: '/qwerty/uyt' is not a valid FID PASS 29a (2s) == sanity-hsm test 29b: Archive/delete/remove by FID from the archive. ========================================================== 01:01:20 (1780808480) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_29b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530476 s, 19.8 MB/s Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' PASS 29b (6s) == sanity-hsm test 29c: Archive/delete/remove by FID, using a file list. ========================================================== 01:01:26 (1780808486) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_29c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489848 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576928 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532844 s, 19.7 MB/s PASS 29c (5s) == sanity-hsm test 29d: hsm_remove by FID with archive_id 0 for unlinked file cause ========================================================== 01:01:31 (1780808491) SKIP: sanity-hsm test_29d Need 3 or more clients, have 1 SKIP 29d (1s) == sanity-hsm test 30a: Restore at exec (import case) ==== 01:01:32 (1780808492) SKIP: sanity-hsm test_30a Need 2 or more clients, have 1 SKIP 30a (1s) == sanity-hsm test 30b: Restore at exec (release case) === 01:01:33 (1780808493) SKIP: sanity-hsm test_30b Need 2 or more clients, have 1 SKIP 30b (1s) == sanity-hsm test 30c: Update during exec of released file must fail ========================================================== 01:01:34 (1780808494) SKIP: sanity-hsm test_30c Need 2 or more clients, have 1 SKIP 30c (1s) == sanity-hsm test 31a: Import a large file and check size during restore ========================================================== 01:01:35 (1780808495) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_31a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 39+0 records in 39+0 records out 40894464 bytes (41 MB, 39 MiB) copied, 0.415249 s, 98.5 MB/s lhsmtool_posix: 1780808496.800760 [60154]: action=1 src=d31a.sanity-hsm/f31a.sanity-hsm dst=/mnt/lustre/d31a.sanity-hsm/f31a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1780808496.803528 [60154]: importing '/mnt/lustre/d31a.sanity-hsm/f31a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_31a//d31a.sanity-hsm/f31a.sanity-hsm' lhsmtool_posix: 1780808496.816717 [60154]: imported '/mnt/lustre/d31a.sanity-hsm/f31a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_31a//040b/0x200000407:0xc:0x0'=='/tmp/arc1/sanity-hsm.test_31a//d31a.sanity-hsm/f31a.sanity-hsm' lhsmtool_posix: 1780808496.816767 [60154]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_31a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' size seen is right: 40894464 == 40894464 done PASS 31a (13s) == sanity-hsm test 31b: Restore a large unaligned file and check size during restore ========================================================== 01:01:49 (1780808509) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_31b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 39+0 records in 39+0 records out 39000000 bytes (39 MB, 37 MiB) copied, 0.326642 s, 119 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' size seen is right: 39000000 == 39000000 done PASS 31b (14s) == sanity-hsm test 31c: Restore a large aligned file and check size during restore ========================================================== 01:02:03 (1780808523) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_31c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 39+0 records in 39+0 records out 40894464 bytes (41 MB, 39 MiB) copied, 0.33984 s, 120 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' size seen is right: 40894464 == 40894464 done PASS 31c (15s) == sanity-hsm test 33: Kill a restore waiting process ==== 01:02:18 (1780808538) 0+0 records in 0+0 records out 0 bytes copied, 0.0015012 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_33/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Copytool is suspended on oleg406-client.virtnet Waiting 200s for 'STARTED' Copytool is continued on oleg406-client.virtnet /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh: line 2893: 62182 Terminated md5sum $f > /dev/null PASS 33 (4s) == sanity-hsm test 34: Remove file during restore ======== 01:02:23 (1780808543) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_34/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes copied, 0.00111902 s, 0.0 kB/s Waiting 200s for 'SUCCEED' Copytool is suspended on oleg406-client.virtnet Waiting 200s for 'STARTED' Copytool is continued on oleg406-client.virtnet /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh: line 2925: kill: (62857) - No such process PASS 34 (5s) == sanity-hsm test 35: Overwrite file during restore ===== 01:02:29 (1780808549) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_35/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes copied, 0.00172183 s, 0.0 kB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Copytool is suspended on oleg406-client.virtnet Waiting 200s for 'STARTED' Copytool is continued on oleg406-client.virtnet /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh: line 2962: kill: (63550) - No such process PASS 35 (6s) == sanity-hsm test 36: Move file during restore ========== 01:02:35 (1780808555) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_36/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes copied, 0.00161686 s, 0.0 kB/s Copytool is suspended on oleg406-client.virtnet Waiting 200s for 'STARTED' Updated after 2s: want 'STARTED' got 'STARTED' Copytool is continued on oleg406-client.virtnet /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh: line 2998: kill: (64232) - No such process PASS 36 (5s) == sanity-hsm test 37: re-archive a dirty file =========== 01:02:40 (1780808560) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_37/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054366 s, 19.3 MB/s Waiting 200s for 'SUCCEED' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204837 s, 51.2 MB/s Waiting 200s for 'SUCCEED' Changed after 1s: from 'SUCCEED WAITING' to 'SUCCEED SUCCEED' Updated after 5s: want 'SUCCEED' got 'SUCCEED' PASS 37 (71s) == sanity-hsm test 40: Parallel archive requests ========= 01:03:51 (1780808631) oleg406-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/hsm/max_requests=1152921504606846976: Cannot allocate memory oleg406-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/max_requests'='1152921504606846976': Cannot allocate memory pdsh@oleg406-client: oleg406-server: ssh exited with exit code 12 oleg406-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0001/hsm/max_requests=1152921504606846976: Cannot allocate memory oleg406-server: error: set_param: setting 'mdt/lustre-MDT0001/hsm/max_requests'='1152921504606846976': Cannot allocate memory pdsh@oleg406-client: oleg406-server: ssh exited with exit code 12 set max_requests=1152921504606846976 failed with 12 [ 895.304575] Lustre: lustre-MDT0000: No more memory to set HSM max_requests=1152921504606846976 (max request memory: 461MB, current total 6/975941): rc = -12 Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_40/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge Wait for all requests being enqueued...100 archive requests submitted 100 archive requests submitted 100 archive requests submitted 100 archive requests submitted OK pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 PASS 40 (87s) == sanity-hsm test 50: Archive with large number of pending HSM actions ========================================================== 01:05:19 (1780808719) oleg406-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/hsm/max_requests=1000000: Cannot allocate memory oleg406-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/max_requests'='1000000': Cannot allocate memory pdsh@oleg406-client: oleg406-server: ssh exited with exit code 12 oleg406-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0001/hsm/max_requests=1000000: Cannot allocate memory oleg406-server: error: set_param: setting 'mdt/lustre-MDT0001/hsm/max_requests'='1000000': Cannot allocate memory pdsh@oleg406-client: oleg406-server: ssh exited with exit code 12 Filesystem Inodes IUsed IFree IUse% Mounted on 192.168.204.106@tcp:/lustre 236391 1319 235072 1% /mnt/lustre - create 8685 (time 1780808731.15 total 10.00 last 868.42) total: 10000 create in 11.73 seconds: 852.32 ops/second 10000 - unlinked 0 (time 1780808754 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second Start Phase files_num: 10000 time: 19 - create 8458 (time 1780808778.18 total 10.08 last 839.45) - create 17840 (time 1780808788.19 total 20.08 last 938.13) total: 20000 create in 22.67 seconds: 882.27 ops/second - unlinked 0 (time 1780808832 ; total 0 ; last 0) - unlinked 10000 (time 1780808844 ; total 12 ; last 12) total: 20000 unlinks in 24 seconds: 833.333313 unlinks/second Middle Phase files_num: 20000 time: 40 - create 8875 (time 1780808867.92 total 10.00 last 887.45) total: 10000 create in 11.34 seconds: 881.70 ops/second 40000 - unlinked 0 (time 1780808891 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second End Phase files_num: 10000 time: 21 40000 mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge PASS 50 (191s) == sanity-hsm test 52: Opened for write file on an evicted client should be set dirty ========================================================== 01:08:30 (1780808910) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_52/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822366 s, 12.8 MB/s multiop /mnt/lustre/d52.sanity-hsm/f52.sanity-hsm vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8369 PASS 52 (5s) == sanity-hsm test 53: Opened for read file on an evicted client should not be set dirty ========================================================== 01:08:35 (1780808915) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_53/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519109 s, 20.2 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' multiop /mnt/lustre/d53.sanity-hsm/f53.sanity-hsm vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.8369 pdsh@oleg406-client: oleg406-client: ssh exited with exit code 5 PASS 53 (8s) == sanity-hsm test 54: Write during an archive cancels it ========================================================== 01:08:43 (1780808923) 39+0 records in 39+0 records out 39000000 bytes (39 MB, 37 MiB) copied, 0.30397 s, 128 MB/s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_54/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' Waiting 200s for 'STARTED' mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0001.hsm.policy=+NRA Waiting 200s for 'FAILED' Waiting 170s for 'FAILED' Updated after 33s: want 'FAILED' got 'FAILED' mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA PASS 54 (40s) == sanity-hsm test 55: Truncate during an archive cancels it ========================================================== 01:09:23 (1780808963) 39+0 records in 39+0 records out 39000000 bytes (39 MB, 37 MiB) copied, 0.314587 s, 124 MB/s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_55/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0001.hsm.policy=+NRA mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA PASS 55 (6s) == sanity-hsm test 56: Setattr during an archive is ok === 01:09:29 (1780808969) 39+0 records in 39+0 records out 39000000 bytes (39 MB, 37 MiB) copied, 0.29512 s, 132 MB/s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_56/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Waiting 190s for 'SUCCEED' Waiting 180s for 'SUCCEED' Waiting 170s for 'SUCCEED' Updated after 38s: want 'SUCCEED' got 'SUCCEED' PASS 56 (41s) == sanity-hsm test 57: Archive a file with dirty cache on another node ========================================================== 01:10:10 (1780809010) SKIP: sanity-hsm test_57 Need 2 or more clients, have 1 SKIP 57 (1s) == sanity-hsm test 58: Truncate a released file will trigger restore ========================================================== 01:10:11 (1780809011) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_58/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' truncate up from 1516 to 3032 /mnt/lustre/d58.sanity-hsm/f58.sanity-hsm: (0x0000000b) exists dirty archived, archive_id:2 truncate down from 1516 to 758 Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' /mnt/lustre/d58.sanity-hsm/f58.sanity-hsm: (0x0000000b) exists dirty archived, archive_id:2 truncate to 0 Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' /mnt/lustre/d58.sanity-hsm/f58.sanity-hsm: (0x0000000b) exists dirty archived, archive_id:2 PASS 58 (10s) == sanity-hsm test 59: Release stripeless file with non-zero size ========================================================== 01:10:21 (1780809021) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_59/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' PASS 59 (4s) == sanity-hsm test 60: Changing progress update interval from default ========================================================== 01:10:25 (1780809025) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0786091 s, 133 MB/s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_60/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 --update-interval 5 "/mnt/lustre2"' Waiting 100s for 'start copy' Expecting a progress update within 20 seconds... 5242880 bytes copied in 4 seconds. Wait for on going archive hsm action to complete Waiting 10s for 'copied' Updated after 5s: want 'copied' got 'copied' PASS 60 (13s) == sanity-hsm test 61: Waiting archive of a removed file should fail ========================================================== 01:10:38 (1780809038) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_61/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 61 (4s) == sanity-hsm test 70: Copytool logs JSON register/unregister events to FIFO ========================================================== 01:10:43 (1780809043) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_70/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --event-fifo /tmp/sanity-hsm.test_70.6qf9/fifo "/mnt/lustre2"' Killing existing copytools on oleg406-client.virtnet with signal TERM copytools stopped in 0s Register/Unregister events look OK. PASS 70 (3s) == sanity-hsm test 71: Copytool logs JSON archive events to FIFO ========================================================== 01:10:47 (1780809047) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_71/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --update-interval 5 --event-fifo /tmp/sanity-hsm.test_71.yID8/fifo "/mnt/lustre2"' mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502763 s, 20.9 MB/s Waiting 200s for 'SUCCEED' Archive events look OK. mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA PASS 71 (8s) == sanity-hsm test 72: Copytool logs JSON restore events to FIFO ========================================================== 01:10:55 (1780809055) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_72/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --update-interval 5 --event-fifo /tmp/sanity-hsm.test_72.Gik9/fifo "/mnt/lustre2"' 16+0 records in 16+0 records out 16000000 bytes (16 MB, 15 MiB) copied, 0.186176 s, 85.9 MB/s lhsmtool_posix: 1780809057.385828 [82067]: action=1 src=d72.sanity-hsm/f72.sanity-hsm dst=/mnt/lustre/d72.sanity-hsm/f72.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1780809057.388778 [82067]: importing '/mnt/lustre/d72.sanity-hsm/f72.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_72//d72.sanity-hsm/f72.sanity-hsm' lhsmtool_posix: 1780809057.402636 [82067]: imported '/mnt/lustre/d72.sanity-hsm/f72.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_72//0bc4/0x200000bd2:0x16:0x0'=='/tmp/arc1/sanity-hsm.test_72//d72.sanity-hsm/f72.sanity-hsm' lhsmtool_posix: 1780809057.402679 [82067]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Verifying released state: Waiting 200s for 'SUCCEED' Restore events look OK. PASS 72 (6s) == sanity-hsm test 90: Archive/restore a file list ======= 01:11:01 (1780809061) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_90/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge Waiting 200s for '' pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Waiting 200s for '' Changed after 2s: from 'lrh=[type=10680000 len=136 idx=4/10058] fid=[0x200000403:0x267:0x0] dfid=[0x200000403:0x267:0x0] compound/cookie=0x0/0x6a2597cb action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10059] fid=[0x200000403:0x268:0x0] dfid=[0x200000403:0x268:0x0] compound/cookie=0x0/0x6a2597cc action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10060] fid=[0x200000403:0x269:0x0] dfid=[0x200000403:0x269:0x0] compound/cookie=0x0/0x6a2597cd action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10061] fid=[0x200000403:0x26a:0x0] dfid=[0x200000403:0x26a:0x0] compound/cookie=0x0/0x6a2597ce action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10062] fid=[0x200000403:0x26b:0x0] dfid=[0x200000403:0x26b:0x0] compound/cookie=0x0/0x6a2597cf action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10063] fid=[0x200000403:0x26c:0x0] dfid=[0x200000403:0x26c:0x0] compound/cookie=0x0/0x6a2597d0 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10064] fid=[0x200000403:0x26d:0x0] dfid=[0x200000403:0x26d:0x0] compound/cookie=0x0/0x6a2597d1 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10066] fid=[0x200000403:0x26e:0x0] dfid=[0x200000403:0x26e:0x0] compound/cookie=0x0/0x6a2597d2 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10067] fid=[0x200000403:0x26f:0x0] dfid=[0x200000403:0x26f:0x0] compound/cookie=0x0/0x6a2597d3 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10068] fid=[0x200000403:0x270:0x0] dfid=[0x200000403:0x270:0x0] compound/cookie=0x0/0x6a2597d4 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10069] fid=[0x200000403:0x271:0x0] dfid=[0x200000403:0x271:0x0] compound/cookie=0x0/0x6a2597d5 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10070] fid=[0x200000403:0x272:0x0] dfid=[0x200000403:0x272:0x0] compound/cookie=0x0/0x6a2597d6 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10071] fid=[0x200000403:0x273:0x0] dfid=[0x200000403:0x273:0x0] compound/cookie=0x0/0x6a2597d7 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10072] fid=[0x200000403:0x274:0x0] dfid=[0x200000403:0x274:0x0] compound/cookie=0x0/0x6a2597d8 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10073] fid=[0x200000403:0x275:0x0] dfid=[0x200000403:0x275:0x0] compound/cookie=0x0/0x6a2597d9 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10074] fid=[0x200000403:0x276:0x0] dfid=[0x200000403:0x276:0x0] compound/cookie=0x0/0x6a2597da action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10075] fid=[0x200000403:0x277:0x0] dfid=[0x200000403:0x277:0x0] compound/cookie=0x0/0x6a2597db action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10076] fid=[0x200000403:0x278:0x0] dfid=[0x200000403:0x278:0x0] compound/cookie=0x0/0x6a2597dc action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10077] fid=[0x200000403:0x279:0x0] dfid=[0x200000403:0x279:0x0] compound/cookie=0x0/0x6a2597dd action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10078] fid=[0x200000403:0x27a:0x0] dfid=[0x200000403:0x27a:0x0] compound/cookie=0x0/0x6a2597de action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10079] fid=[0x200000403:0x27b:0x0] dfid=[0x200000403:0x27b:0x0] compound/cookie=0x0/0x6a2597df action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10080] fid=[0x200000403:0x27c:0x0] dfid=[0x200000403:0x27c:0x0] compound/cookie=0x0/0x6a2597e0 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10081] fid=[0x200000403:0x27d:0x0] dfid=[0x200000403:0x27d:0x0] compound/cookie=0x0/0x6a2597e1 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10082] fid=[0x200000403:0x27e:0x0] dfid=[0x200000403:0x27e:0x0] compound/cookie=0x0/0x6a2597e2 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10083] fid=[0x200000403:0x27f:0x0] dfid=[0x200000403:0x27f:0x0] compound/cookie=0x0/0x6a2597e3 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10084] fid=[0x200000403:0x280:0x0] dfid=[0x200000403:0x280:0x0] compound/cookie=0x0/0x6a2597e4 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10085] fid=[0x200000403:0x281:0x0] dfid=[0x200000403:0x281:0x0] compound/cookie=0x0/0x6a2597e5 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10086] fid=[0x200000403:0x282:0x0] dfid=[0x200000403:0x282:0x0] compound/cookie=0x0/0x6a2597e6 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10087] fid=[0x200000403:0x283:0x0] dfid=[0x200000403:0x283:0x0] compound/cookie=0x0/0x6a2597e7 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10088] fid=[0x200000403:0x284:0x0] dfid=[0x200000403:0x284:0x0] compound/cookie=0x0/0x6a2597e8 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10089] fid=[0x200000403:0x285:0x0] dfid=[0x200000403:0x285:0x0] compound/cookie=0x0/0x6a2597e9 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10090] fid=[0x200000403:0x286:0x0] dfid=[0x200000403:0x286:0x0] compound/cookie=0x0/0x6a2597ea action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10091] fid=[0x200000403:0x287:0x0] dfid=[0x200000403:0x287:0x0] compound/cookie=0x0/0x6a2597eb action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10092] fid=[0x200000403:0x288:0x0] dfid=[0x200000403:0x288:0x0] compound/cookie=0x0/0x6a2597ec action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10093] fid=[0x200000403:0x289:0x0] dfid=[0x200000403:0x289:0x0] compound/cookie=0x0/0x6a2597ed action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10094] fid=[0x200000403:0x28a:0x0] dfid=[0x200000403:0x28a:0x0] compound/cookie=0x0/0x6a2597ee action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10095] fid=[0x200000403:0x28b:0x0] dfid=[0x200000403:0x28b:0x0] compound/cookie=0x0/0x6a2597ef action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10096] fid=[0x200000403:0x28c:0x0] dfid=[0x200000403:0x28c:0x0] compound/cookie=0x0/0x6a2597f0 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10097] fid=[0x200000403:0x28d:0x0] dfid=[0x200000403:0x28d:0x0] compound/cookie=0x0/0x6a2597f1 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10098] fid=[0x200000403:0x28e:0x0] dfid=[0x200000403:0x28e:0x0] compound/cookie=0x0/0x6a2597f2 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10099] fid=[0x200000403:0x28f:0x0] dfid=[0x200000403:0x28f:0x0] compound/cookie=0x0/0x6a2597f3 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10100] fid=[0x200000403:0x290:0x0] dfid=[0x200000403:0x290:0x0] compound/cookie=0x0/0x6a2597f4 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10101] fid=[0x200000403:0x291:0x0] dfid=[0x200000403:0x291:0x0] compound/cookie=0x0/0x6a2597f5 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10102] fid=[0x200000403:0x292:0x0] dfid=[0x200000403:0x292:0x0] compound/cookie=0x0/0x6a2597f6 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10103] fid=[0x200000403:0x293:0x0] dfid=[0x200000403:0x293:0x0] compound/cookie=0x0/0x6a2597f7 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10104] fid=[0x200000403:0x294:0x0] dfid=[0x200000403:0x294:0x0] compound/cookie=0x0/0x6a2597f8 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10105] fid=[0x200000403:0x295:0x0] dfid=[0x200000403:0x295:0x0] compound/cookie=0x0/0x6a2597f9 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10106] fid=[0x200000403:0x296:0x0] dfid=[0x200000403:0x296:0x0] compound/cookie=0x0/0x6a2597fa action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10107] fid=[0x200000403:0x297:0x0] dfid=[0x200000403:0x297:0x0] compound/cookie=0x0/0x6a2597fb action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10108] fid=[0x200000403:0x298:0x0] dfid=[0x200000403:0x298:0x0] compound/cookie=0x0/0x6a2597fc action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10109] fid=[0x200000403:0x299:0x0] dfid=[0x200000403:0x299:0x0] compound/cookie=0x0/0x6a2597fd action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[]' to 'lrh=[type=10680000 len=136 idx=4/10087] fid=[0x200000403:0x283:0x0] dfid=[0x200000403:0x283:0x0] compound/cookie=0x0/0x6a2597e7 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[] lrh=[type=10680000 len=136 idx=4/10089] fid=[0x200000403:0x285:0x0] dfid=[0x200000403:0x285:0x0] compound/cookie=0x0/0x6a2597e9 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10090] fid=[0x200000403:0x286:0x0] dfid=[0x200000403:0x286:0x0] compound/cookie=0x0/0x6a2597ea action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10091] fid=[0x200000403:0x287:0x0] dfid=[0x200000403:0x287:0x0] compound/cookie=0x0/0x6a2597eb action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10092] fid=[0x200000403:0x288:0x0] dfid=[0x200000403:0x288:0x0] compound/cookie=0x0/0x6a2597ec action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10093] fid=[0x200000403:0x289:0x0] dfid=[0x200000403:0x289:0x0] compound/cookie=0x0/0x6a2597ed action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10094] fid=[0x200000403:0x28a:0x0] dfid=[0x200000403:0x28a:0x0] compound/cookie=0x0/0x6a2597ee action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10095] fid=[0x200000403:0x28b:0x0] dfid=[0x200000403:0x28b:0x0] compound/cookie=0x0/0x6a2597ef action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10096] fid=[0x200000403:0x28c:0x0] dfid=[0x200000403:0x28c:0x0] compound/cookie=0x0/0x6a2597f0 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10097] fid=[0x200000403:0x28d:0x0] dfid=[0x200000403:0x28d:0x0] compound/cookie=0x0/0x6a2597f1 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10098] fid=[0x200000403:0x28e:0x0] dfid=[0x200000403:0x28e:0x0] compound/cookie=0x0/0x6a2597f2 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10099] fid=[0x200000403:0x28f:0x0] dfid=[0x200000403:0x28f:0x0] compound/cookie=0x0/0x6a2597f3 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10100] fid=[0x200000403:0x290:0x0] dfid=[0x200000403:0x290:0x0] compound/cookie=0x0/0x6a2597f4 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10101] fid=[0x200000403:0x291:0x0] dfid=[0x200000403:0x291:0x0] compound/cookie=0x0/0x6a2597f5 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10102] fid=[0x200000403:0x292:0x0] dfid=[0x200000403:0x292:0x0] compound/cookie=0x0/0x6a2597f6 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10103] fid=[0x200000403:0x293:0x0] dfid=[0x200000403:0x293:0x0] compound/cookie=0x0/0x6a2597f7 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10104] fid=[0x200000403:0x294:0x0] dfid=[0x200000403:0x294:0x0] compound/cookie=0x0/0x6a2597f8 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10105] fid=[0x200000403:0x295:0x0] dfid=[0x200000403:0x295:0x0] compound/cookie=0x0/0x6a2597f9 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10106] fid=[0x200000403:0x296:0x0] dfid=[0x200000403:0x296:0x0] compound/cookie=0x0/0x6a2597fa action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10107] fid=[0x200000403:0x297:0x0] dfid=[0x200000403:0x297:0x0] compound/cookie=0x0/0x6a2597fb action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10108] fid=[0x200000403:0x298:0x0] dfid=[0x200000403:0x298:0x0] compound/cookie=0x0/0x6a2597fc action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/10109] fid=[0x200000403:0x299:0x0] dfid=[0x200000403:0x299:0x0] compound/cookie=0x0/0x6a2597fd action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[]' pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 PASS 90 (72s) == sanity-hsm test 100: Set coordinator /proc tunables === 01:12:13 (1780809133) Testing mdt.lustre-MDT0000.hsm.loop_period Set loop_period to 0 must failed oleg406-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/hsm/loop_period=0: Invalid argument oleg406-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/loop_period'='0': Invalid argument pdsh@oleg406-client: oleg406-server: ssh exited with exit code 22 oleg406-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0001/hsm/loop_period=0: Invalid argument oleg406-server: error: set_param: setting 'mdt/lustre-MDT0001/hsm/loop_period'='0': Invalid argument pdsh@oleg406-client: oleg406-server: ssh exited with exit code 22 Testing mdt.lustre-MDT0000.hsm.grace_delay Set grace_delay to 0 must failed oleg406-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/hsm/grace_delay=0: Invalid argument oleg406-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/grace_delay'='0': Invalid argument pdsh@oleg406-client: oleg406-server: ssh exited with exit code 22 oleg406-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0001/hsm/grace_delay=0: Invalid argument oleg406-server: error: set_param: setting 'mdt/lustre-MDT0001/hsm/grace_delay'='0': Invalid argument pdsh@oleg406-client: oleg406-server: ssh exited with exit code 22 Testing mdt.lustre-MDT0000.hsm.active_request_timeout Set active_request_timeout to 0 must failed oleg406-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/hsm/active_request_timeout=0: Invalid argument oleg406-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/active_request_timeout'='0': Invalid argument pdsh@oleg406-client: oleg406-server: ssh exited with exit code 22 oleg406-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0001/hsm/active_request_timeout=0: Invalid argument oleg406-server: error: set_param: setting 'mdt/lustre-MDT0001/hsm/active_request_timeout'='0': Invalid argument pdsh@oleg406-client: oleg406-server: ssh exited with exit code 22 Testing mdt.lustre-MDT0000.hsm.max_requests Set max_requests to 0 must failed oleg406-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/hsm/max_requests=0: Invalid argument oleg406-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/max_requests'='0': Invalid argument pdsh@oleg406-client: oleg406-server: ssh exited with exit code 22 oleg406-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0001/hsm/max_requests=0: Invalid argument oleg406-server: error: set_param: setting 'mdt/lustre-MDT0001/hsm/max_requests'='0': Invalid argument pdsh@oleg406-client: oleg406-server: ssh exited with exit code 22 Testing mdt.lustre-MDT0000.hsm.default_archive_id Set default_archive_id to 0 must failed oleg406-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/hsm/default_archive_id=0: Invalid argument oleg406-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/default_archive_id'='0': Invalid argument pdsh@oleg406-client: oleg406-server: ssh exited with exit code 22 oleg406-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0001/hsm/default_archive_id=0: Invalid argument oleg406-server: error: set_param: setting 'mdt/lustre-MDT0001/hsm/default_archive_id'='0': Invalid argument pdsh@oleg406-client: oleg406-server: ssh exited with exit code 22 PASS 100 (17s) == sanity-hsm test 102: Verify coordinator control ======= 01:12:30 (1780809150) mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 102 (6s) == sanity-hsm test 103: Purge all requests =============== 01:12:36 (1780809156) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_103/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge Current requests pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 PASS 103 (6s) == sanity-hsm test 103a: Purge pending restore requests == 01:12:42 (1780809162) mdt.lustre-MDT0000.hsm.policy=-NBR mdt.lustre-MDT0001.hsm.policy=-NBR Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_103a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' (0x200000403:0x2e1:0x0|0x200000403:0x2e2:0x0|0x200000403:0x2e3:0x0|0x200000403:0x2e4:0x0|0x200000403:0x2e5:0x0|0x200000403:0x2e6:0x0|0x200000403:0x2e7:0x0|0x200000403:0x2e8:0x0).*action=ARCHIVE.*status=SUCCEED pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Killing existing copytools on oleg406-client.virtnet with signal TERM copytools stopped in 0s cat: '/mnt/lustre/.lustre/fid/0x200000403:0x2e1:0x0': No data available cat: '/mnt/lustre/.lustre/fid/0x200000403:0x2e2:0x0': No data available cat: '/mnt/lustre/.lustre/fid/0x200000403:0x2e4:0x0': No data available cat: '/mnt/lustre/.lustre/fid/0x200000403:0x2e3:0x0': No data available cat: '/mnt/lustre/.lustre/fid/0x200000403:0x2e5:0x0': No data available cat: '/mnt/lustre/.lustre/fid/0x200000403:0x2e6:0x0': No data available cat: '/mnt/lustre/.lustre/fid/0x200000403:0x2e7:0x0': No data available cat: '/mnt/lustre/.lustre/fid/0x200000403:0x2e8:0x0': No data available mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge PASS 103a (8s) == sanity-hsm test 104: Copy tool data field ============= 01:12:50 (1780809170) 0+0 records in 0+0 records out 0 bytes copied, 0.0014459 s, 0.0 kB/s mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge PASS 104 (5s) == sanity-hsm test 105: Restart of coordinator =========== 01:12:55 (1780809175) mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge PASS 105 (11s) == sanity-hsm test 106a: Copytool register/unregister ==== 01:13:06 (1780809186) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_106a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' oleg406-client.virtnet: executing pgrep --pidfile=/var/run/lhsmtool_posix.pid --list-full hsmtool oleg406-client.virtnet: executing get_client_uuid /mnt/lustre2 found agent uuid=c90ca412-7c0e-430e-8b00-175241015fe3 archive_id=ANY requests=[current:0 ok:0 errors:0] on mds1 found agent uuid=c90ca412-7c0e-430e-8b00-175241015fe3 archive_id=ANY requests=[current:0 ok:0 errors:0] on mds2 oleg406-client.virtnet: 88752 Killing existing copytools on oleg406-client.virtnet with signal TERM copytools stopped in 0s pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 uuid not found in agent list on mds1 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 uuid not found in agent list on mds2 Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_106a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' oleg406-client.virtnet: executing pgrep --pidfile=/var/run/lhsmtool_posix.pid --list-full hsmtool oleg406-client.virtnet: executing get_client_uuid /mnt/lustre2 found agent uuid=c90ca412-7c0e-430e-8b00-175241015fe3 archive_id=ANY requests=[current:0 ok:0 errors:0] on mds1 found agent uuid=c90ca412-7c0e-430e-8b00-175241015fe3 archive_id=ANY requests=[current:0 ok:0 errors:0] on mds2 PASS 106a (9s) == sanity-hsm test 106b: Unregister agent after a copytool crash ========================================================== 01:13:15 (1780809195) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_106b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' oleg406-client.virtnet: executing pgrep --pidfile=/var/run/lhsmtool_posix.pid --list-full hsmtool oleg406-client.virtnet: executing get_client_uuid /mnt/lustre2 found agent uuid=c90ca412-7c0e-430e-8b00-175241015fe3 archive_id=ANY requests=[current:0 ok:0 errors:0] on mds1 found agent uuid=c90ca412-7c0e-430e-8b00-175241015fe3 archive_id=ANY requests=[current:0 ok:0 errors:0] on mds2 oleg406-client.virtnet: 90400 Killing existing copytools on oleg406-client.virtnet with signal KILL copytools stopped in 0s pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 uuid not found in agent list on mds1 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 uuid not found in agent list on mds2 Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_106b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' found agent uuid=c90ca412-7c0e-430e-8b00-175241015fe3 archive_id=ANY requests=[current:0 ok:0 errors:0] on mds1 found agent uuid=c90ca412-7c0e-430e-8b00-175241015fe3 archive_id=ANY requests=[current:0 ok:0 errors:0] on mds2 PASS 106b (6s) == sanity-hsm test 106c: Unregister copytool on client eviction ========================================================== 01:13:21 (1780809201) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_106c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' oleg406-client.virtnet: executing pgrep --pidfile=/var/run/lhsmtool_posix.pid --list-full hsmtool oleg406-client.virtnet: executing get_client_uuid /mnt/lustre2 found agent uuid=c90ca412-7c0e-430e-8b00-175241015fe3 archive_id=ANY requests=[current:0 ok:0 errors:0] on mds1 found agent uuid=c90ca412-7c0e-430e-8b00-175241015fe3 archive_id=ANY requests=[current:0 ok:0 errors:0] on mds2 oleg406-client.virtnet: 91588 oleg406-client.virtnet: executing pgrep --pidfile=/var/run/lhsmtool_posix.pid --list-full hsmtool oleg406-client.virtnet: executing pgrep --pidfile=/var/run/lhsmtool_posix.pid --list-full hsmtool oleg406-client.virtnet: executing get_client_uuid /mnt/lustre2 mdc.lustre-MDT0000-mdc-ffff90f1831cd800.active=0 mdc.lustre-MDT0001-mdc-ffff90f1831cd800.active=0 mdt.lustre-MDT0000.evict_client=c90ca412-7c0e-430e-8b00-175241015fe3 mdt.lustre-MDT0001.evict_client=c90ca412-7c0e-430e-8b00-175241015fe3 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 uuid not found in agent list on mds1 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 uuid not found in agent list on mds2 mdc.lustre-MDT0000-mdc-ffff90f1831cd800.active=1 mdc.lustre-MDT0001-mdc-ffff90f1831cd800.active=1 oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec found agent uuid=c90ca412-7c0e-430e-8b00-175241015fe3 archive_id=ANY requests=[current:0 ok:0 errors:0] on mds1 found agent uuid=c90ca412-7c0e-430e-8b00-175241015fe3 archive_id=ANY requests=[current:0 ok:0 errors:0] on mds2 mdc.lustre-MDT0000-mdc-ffff90f1831cd800.active=1 mdc.lustre-MDT0001-mdc-ffff90f1831cd800.active=1 PASS 106c (11s) == sanity-hsm test 107: Copytool re-register after MDS restart ========================================================== 01:13:32 (1780809212) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_107/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Failing mds1 on oleg406-server Stopping /mnt/lustre-mds1 (opts:) on oleg406-server 01:13:35 (1780809215) shut down facet: mds1 facet_host: oleg406-server facet_failover_host: oleg406-server Failover mds1 to oleg406-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 01:13:49 (1780809229) targets are mounted 01:13:49 (1780809229) facet_failover done oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 107 (25s) == sanity-hsm test 109: Policy display/change ============ 01:13:57 (1780809237) mdt.lustre-MDT0000.hsm.policy=+NBR mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0000.hsm.policy=-NBR mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0000.hsm.policy=NRA NBR Next set_param must failed oleg406-server: error: set_param: setting /sys/kernel/debug/lustre/mdt/lustre-MDT0000/hsm/policy=wrong: Invalid argument oleg406-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/policy'='wrong': Invalid argument pdsh@oleg406-client: oleg406-server: ssh exited with exit code 22 Back to default policy mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0001.hsm.policy=+NRA mdt.lustre-MDT0000.hsm.policy=-NBR mdt.lustre-MDT0001.hsm.policy=-NBR PASS 109 (7s) == sanity-hsm test 110a: Non blocking restore policy (import case) ========================================================== 01:14:04 (1780809244) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_110a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1780809245.139414 [96025]: action=1 src=d110a.sanity-hsm/f110a.sanity-hsm dst=/mnt/lustre/d110a.sanity-hsm/f110a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1780809245.141401 [96025]: importing '/mnt/lustre/d110a.sanity-hsm/f110a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_110a//d110a.sanity-hsm/f110a.sanity-hsm' lhsmtool_posix: 1780809245.150658 [96025]: imported '/mnt/lustre/d110a.sanity-hsm/f110a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_110a//0bb1/0x200000bd2:0x63:0x0'=='/tmp/arc1/sanity-hsm.test_110a//d110a.sanity-hsm/f110a.sanity-hsm' lhsmtool_posix: 1780809245.150691 [96025]: process finished, errs: 0 major, 0 minor, rc=0 (Success) mdt.lustre-MDT0000.hsm.policy=+NBR mdt.lustre-MDT0001.hsm.policy=+NBR md5sum: /mnt/lustre/d110a.sanity-hsm/f110a.sanity-hsm: No data available Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' mdt.lustre-MDT0000.hsm.policy=-NBR mdt.lustre-MDT0001.hsm.policy=-NBR PASS 110a (5s) == sanity-hsm test 110b: Non blocking restore policy (release case) ========================================================== 01:14:09 (1780809249) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_110b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' mdt.lustre-MDT0000.hsm.policy=+NBR mdt.lustre-MDT0001.hsm.policy=+NBR md5sum: /mnt/lustre/d110b.sanity-hsm/f110b.sanity-hsm: No data available Waiting 200s for 'SUCCEED' mdt.lustre-MDT0000.hsm.policy=-NBR mdt.lustre-MDT0001.hsm.policy=-NBR PASS 110b (7s) == sanity-hsm test 111a: No retry policy (import case), restore will error ========================================================== 01:14:16 (1780809256) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_111a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1780809256.770240 [97436]: action=1 src=d111a.sanity-hsm/f111a.sanity-hsm dst=/mnt/lustre/d111a.sanity-hsm/f111a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1780809256.772851 [97436]: importing '/mnt/lustre/d111a.sanity-hsm/f111a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_111a//d111a.sanity-hsm/f111a.sanity-hsm' lhsmtool_posix: 1780809256.782742 [97436]: imported '/mnt/lustre/d111a.sanity-hsm/f111a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_111a//0bb5/0x200000bd2:0x67:0x0'=='/tmp/arc1/sanity-hsm.test_111a//d111a.sanity-hsm/f111a.sanity-hsm' lhsmtool_posix: 1780809256.782788 [97436]: process finished, errs: 0 major, 0 minor, rc=0 (Success) mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0001.hsm.policy=+NRA Remove from backend: 0x200000bd2:0x67:0x0 = /tmp/arc1/sanity-hsm.test_111a/0bb5/0x200000bd2:0x67:0x0 Waiting 200s for 'FAILED' mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA PASS 111a (5s) == sanity-hsm test 111b: No retry policy (release case), restore will error ========================================================== 01:14:21 (1780809261) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_111b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0001.hsm.policy=+NRA Waiting 200s for 'SUCCEED' Remove from backend: 0x200000bd3:0x7:0x0 = /tmp/arc1/sanity-hsm.test_111b/0bd4/0x200000bd3:0x7:0x0 Waiting 200s for 'FAILED' mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA PASS 111b (6s) == sanity-hsm test 112: State of recorded request ======== 01:14:27 (1780809267) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_112/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled /mnt/lustre/d112.sanity-hsm/f112.sanity-hsm: ARCHIVE waiting (from 0 to EOF) mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 112 (5s) == sanity-hsm test 113: wrong stat after restore ========= 01:14:32 (1780809272) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200758 s, 52.2 MB/s Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre3 Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_113/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre3"' Waiting 200s for 'SUCCEED' Verifying released state: multiop /mnt/lustre/d113.sanity-hsm/f113.sanity-hsm voO_WRONLY:O_APPEND:_w4c TMPPIPE=/tmp/multiop_open_wait_pipe.8369 File: /mnt/lustre2/d113.sanity-hsm/f113.sanity-hsm Size: 1048576 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238860488715 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-07 01:14:32.000000000 -0400 Modify: 2026-06-07 01:14:32.000000000 -0400 Change: 2026-06-07 01:14:32.000000000 -0400 Birth: 2026-06-07 01:14:32.000000000 -0400 Stopping client oleg406-client.virtnet /mnt/lustre3 (opts:) PASS 113 (5s) == sanity-hsm test 114: Incompatible request does not set other requests as STARTED ========================================================== 01:14:37 (1780809277) 0+0 records in 0+0 records out 0 bytes copied, 0.00139009 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00108072 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_114/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 114 (5s) == sanity-hsm test 200: Register/Cancel archive ========== 01:14:43 (1780809283) 0+0 records in 0+0 records out 0 bytes copied, 0.00120967 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_200/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Copytool is suspended on oleg406-client.virtnet Waiting 200s for 'CANCELED' Copytool is continued on oleg406-client.virtnet PASS 200 (4s) == sanity-hsm test 201: Register/Cancel restore ========== 01:14:47 (1780809287) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_201/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 39+0 records in 39+0 records out 40894464 bytes (41 MB, 39 MiB) copied, 0.404781 s, 101 MB/s lhsmtool_posix: 1780809288.808355 [101844]: action=1 src=d201.sanity-hsm/f201.sanity-hsm dst=/mnt/lustre/d201.sanity-hsm/f201.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1780809288.810878 [101844]: importing '/mnt/lustre/d201.sanity-hsm/f201.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_201//d201.sanity-hsm/f201.sanity-hsm' lhsmtool_posix: 1780809288.825176 [101844]: imported '/mnt/lustre/d201.sanity-hsm/f201.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_201//0ba3/0x200000bd2:0x71:0x0'=='/tmp/arc1/sanity-hsm.test_201//d201.sanity-hsm/f201.sanity-hsm' lhsmtool_posix: 1780809288.825235 [101844]: process finished, errs: 0 major, 0 minor, rc=0 (Success) mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 201 (6s) == sanity-hsm test 202: Register/Cancel remove =========== 01:14:53 (1780809293) 0+0 records in 0+0 records out 0 bytes copied, 0.0013504 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_202/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Copytool is suspended on oleg406-client.virtnet Waiting 200s for 'CANCELED' PASS 202 (6s) == sanity-hsm test 220A: Changelog for archive =========== 01:14:59 (1780809299) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_220A/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' lustre-MDT0000.3 16HSM 05:15:01.892350696 2026.06.07 0x0 t=[0x200000bd3:0x13:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 PASS 220A (7s) == sanity-hsm test 220a: Changelog for failed archive ==== 01:15:06 (1780809306) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_220a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl2 cl2' Copytool is suspended on oleg406-client.virtnet Copytool is continued on oleg406-client.virtnet lustre-MDT0000.6 16HSM 05:15:09.245347443 2026.06.07 0x2 t=[0x200000bd3:0x14:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0001: Deregistered changelog user #2 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 PASS 220a (7s) == sanity-hsm test 221: Changelog for archive canceled === 01:15:13 (1780809313) 0+0 records in 0+0 records out 0 bytes copied, 0.00200329 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_221/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl3 cl3' Copytool is suspended on oleg406-client.virtnet Waiting 200s for 'STARTED' Waiting 200s for 'CANCELED' Updated after 2s: want 'CANCELED' got 'CANCELED' Copytool is continued on oleg406-client.virtnet lustre-MDT0000.8 16HSM 05:15:19.085142633 2026.06.07 0x7d t=[0x200000bd3:0x15:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp lustre-MDT0001: clear the changelog for cl3 of all records lustre-MDT0001: Deregistered changelog user #3 lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0000: Deregistered changelog user #3 PASS 221 (10s) == sanity-hsm test 222a: Changelog for explicit restore == 01:15:23 (1780809323) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_222a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1780809324.243190 [105714]: action=1 src=d222a.sanity-hsm/f222a.sanity-hsm dst=/mnt/lustre/d222a.sanity-hsm/f222a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1780809324.245991 [105714]: importing '/mnt/lustre/d222a.sanity-hsm/f222a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_222a//d222a.sanity-hsm/f222a.sanity-hsm' lhsmtool_posix: 1780809324.257883 [105714]: imported '/mnt/lustre/d222a.sanity-hsm/f222a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_222a//0ba5/0x200000bd2:0x77:0x0'=='/tmp/arc1/sanity-hsm.test_222a//d222a.sanity-hsm/f222a.sanity-hsm' lhsmtool_posix: 1780809324.257946 [105714]: process finished, errs: 0 major, 0 minor, rc=0 (Success) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl4 cl4' Waiting 200s for 'SUCCEED' lustre-MDT0000.11 16HSM 05:15:26.321653213 2026.06.07 0x80 t=[0x200000bd2:0x77:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp lustre-MDT0001: clear the changelog for cl4 of all records lustre-MDT0001: Deregistered changelog user #4 lustre-MDT0000: clear the changelog for cl4 of all records lustre-MDT0000: Deregistered changelog user #4 PASS 222a (8s) == sanity-hsm test 222b: Changelog for implicit restore == 01:15:31 (1780809331) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_222b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl5 cl5' 4ef076ad20e5d4a2f4729d6ed41a3bf9 /mnt/lustre/d222b.sanity-hsm/f222b.sanity-hsm lustre-MDT0000.20 16HSM 05:15:34.579032002 2026.06.07 0x80 t=[0x200000bd3:0x17:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp lustre-MDT0001: clear the changelog for cl5 of all records lustre-MDT0001: Deregistered changelog user #5 lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0000: Deregistered changelog user #5 PASS 222b (8s) == sanity-hsm test 222c: Changelog for failed explicit restore ========================================================== 01:15:39 (1780809339) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_222c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1780809339.913099 [107348]: action=1 src=d222c.sanity-hsm/f222c.sanity-hsm dst=/mnt/lustre/d222c.sanity-hsm/f222c.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1780809339.915728 [107348]: importing '/mnt/lustre/d222c.sanity-hsm/f222c.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_222c//d222c.sanity-hsm/f222c.sanity-hsm' lhsmtool_posix: 1780809339.927340 [107348]: imported '/mnt/lustre/d222c.sanity-hsm/f222c.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_222c//0bae/0x200000bd2:0x7c:0x0'=='/tmp/arc1/sanity-hsm.test_222c//d222c.sanity-hsm/f222c.sanity-hsm' lhsmtool_posix: 1780809339.927380 [107348]: process finished, errs: 0 major, 0 minor, rc=0 (Success) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl6 cl6' Copytool is suspended on oleg406-client.virtnet Waiting 200s for 'STARTED' Updated after 2s: want 'STARTED' got 'STARTED' Copytool is continued on oleg406-client.virtnet lustre-MDT0000.22 16HSM 05:15:42.925712917 2026.06.07 0x82 t=[0x200000bd2:0x7c:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0001: Deregistered changelog user #6 lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0000: Deregistered changelog user #6 PASS 222c (8s) == sanity-hsm test 222d: Changelog for failed implicit restore ========================================================== 01:15:47 (1780809347) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_222d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl7 cl7' Waiting 200s for 'SUCCEED' Remove from backend: 0x200000bd3:0x19:0x0 = /tmp/arc1/sanity-hsm.test_222d/0bca/0x200000bd3:0x19:0x0 md5sum: /mnt/lustre/d222d.sanity-hsm/f222d.sanity-hsm: No data available lustre-MDT0000.29 16HSM 05:15:51.170578644 2026.06.07 0x82 t=[0x200000bd3:0x19:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp lustre-MDT0001: clear the changelog for cl7 of all records lustre-MDT0001: Deregistered changelog user #7 lustre-MDT0000: clear the changelog for cl7 of all records lustre-MDT0000: Deregistered changelog user #7 PASS 222d (8s) == sanity-hsm test 223a: Changelog for restore canceled (import case) ========================================================== 01:15:55 (1780809355) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_223a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' 39+0 records in 39+0 records out 40894464 bytes (41 MB, 39 MiB) copied, 0.417612 s, 97.9 MB/s mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl8 cl8' lhsmtool_posix: 1780809358.582944 [109102]: action=1 src=d223a.sanity-hsm/f223a.sanity-hsm dst=/mnt/lustre/d223a.sanity-hsm/f223a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1780809358.585931 [109102]: importing '/mnt/lustre/d223a.sanity-hsm/f223a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_223a//d223a.sanity-hsm/f223a.sanity-hsm' lhsmtool_posix: 1780809358.598349 [109102]: imported '/mnt/lustre/d223a.sanity-hsm/f223a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_223a//0b53/0x200000bd2:0x81:0x0'=='/tmp/arc1/sanity-hsm.test_223a//d223a.sanity-hsm/f223a.sanity-hsm' lhsmtool_posix: 1780809358.598470 [109102]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Waiting 200s for 'CANCELED' Updated after 2s: want 'CANCELED' got 'CANCELED' Waiting 200s for 'SUCCEED' Waiting 190s for 'SUCCEED' Updated after 29s: want 'SUCCEED' got 'SUCCEED' lustre-MDT0000.36 16HSM 05:16:27.631372288 2026.06.07 0xfd t=[0x200000bd2:0x81:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp lustre-MDT0001: clear the changelog for cl8 of all records lustre-MDT0001: Deregistered changelog user #8 lustre-MDT0000: clear the changelog for cl8 of all records lustre-MDT0000: Deregistered changelog user #8 PASS 223a (38s) == sanity-hsm test 223b: Changelog for restore canceled (release case) ========================================================== 01:16:33 (1780809393) 0+0 records in 0+0 records out 0 bytes copied, 0.00194871 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_223b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl9 cl9' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Copytool is suspended on oleg406-client.virtnet Waiting 200s for 'STARTED' Copytool is continued on oleg406-client.virtnet lustre-MDT0000.43 16HSM 05:16:39.174095382 2026.06.07 0xfd t=[0x200000bd3:0x1d:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp p=[0x200000bd2:0x7a:0x0] lustre-MDT0001: clear the changelog for cl9 of all records lustre-MDT0001: Deregistered changelog user #9 lustre-MDT0000: clear the changelog for cl9 of all records lustre-MDT0000: Deregistered changelog user #9 PASS 223b (10s) == sanity-hsm test 224A: Changelog for remove ============ 01:16:43 (1780809403) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_224A/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl10 cl10' Waiting 200s for 'SUCCEED' lustre-MDT0000.48 16HSM 05:16:47.476673269 2026.06.07 0x200 t=[0x200000bd3:0x1f:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp p=[0x200000bd2:0x7f:0x0] lustre-MDT0001: clear the changelog for cl10 of all records lustre-MDT0001: Deregistered changelog user #10 lustre-MDT0000: clear the changelog for cl10 of all records lustre-MDT0000: Deregistered changelog user #10 PASS 224A (8s) == sanity-hsm test 224a: Changelog for failed remove ===== 01:16:51 (1780809411) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_224a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl11 cl11' Waiting 200s for 'SUCCEED' Remove from backend: 0x200000bd3:0x20:0x0 = /tmp/arc1/sanity-hsm.test_224a/0bf3/0x200000bd3:0x20:0x0 Copytool is suspended on oleg406-client.virtnet Waiting 200s for 'STARTED' Copytool is continued on oleg406-client.virtnet lustre-MDT0000.53 16HSM 05:16:57.696984404 2026.06.07 0x202 t=[0x200000bd3:0x20:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp lustre-MDT0001: clear the changelog for cl11 of all records lustre-MDT0001: Deregistered changelog user #11 lustre-MDT0000: clear the changelog for cl11 of all records lustre-MDT0000: Deregistered changelog user #11 PASS 224a (11s) == sanity-hsm test 225: Changelog for remove canceled ==== 01:17:02 (1780809422) Test disabled PASS 225 (2s) == sanity-hsm test 226: changelog for last rm/mv with exiting archive ========================================================== 01:17:04 (1780809424) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_226/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 0x200000bd3:0x23:0x0 mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl12 cl12' Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' lustre-MDT0000.54 16HSM 05:17:06.927657625 2026.06.07 0x280 t=[0x200000bd3:0x21:0x0] ef=0x13 u=0:0 nid=0@<0:0> lustre-MDT0000.55 16HSM 05:17:06.949263387 2026.06.07 0x280 t=[0x200000bd3:0x21:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp p=[0x200000bd2:0x7f:0x0] lustre-MDT0000.56 16HSM 05:17:06.949519298 2026.06.07 0x0 t=[0x200000bd3:0x21:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp p=[0x200000bd2:0x7f:0x0] lustre-MDT0000.57 16HSM 05:17:08.975340975 2026.06.07 0x280 t=[0x200000bd3:0x22:0x0] ef=0x13 u=0:0 nid=0@<0:0> lustre-MDT0000.58 16HSM 05:17:09.000872524 2026.06.07 0x280 t=[0x200000bd3:0x22:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp lustre-MDT0000.59 16HSM 05:17:09.001021609 2026.06.07 0x0 t=[0x200000bd3:0x22:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp lustre-MDT0000.60 06UNLNK 05:17:09.866269805 2026.06.07 0x3 t=[0x200000bd3:0x21:0x0] j=rm.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp p=[0x200000bd2:0x86:0x0] f226.sanity-hsm-1 lustre-MDT0000.60 06UNLNK 05:17:09.866269805 2026.06.07 0x3 t=[0x200000bd3:0x21:0x0] j=rm.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp p=[0x200000bd2:0x86:0x0] f226.sanity-hsm-1 lustre-MDT0000.61 08RENME 05:17:10.010655150 2026.06.07 0x3 t=[0x200000bd3:0x22:0x0] j=mv.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp p=[0x200000bd2:0x86:0x0] f226.sanity-hsm-2 s=[0x200000bd3:0x23:0x0] sp=[0x200000bd2:0x86:0x0] f226.sanity-hsm-3 lustre-MDT0001: clear the changelog for cl12 of all records lustre-MDT0001: Deregistered changelog user #12 lustre-MDT0000: clear the changelog for cl12 of all records lustre-MDT0000: Deregistered changelog user #12 PASS 226 (10s) == sanity-hsm test 227: changelog when explicit setting of HSM flags ========================================================== 01:17:14 (1780809434) 0+0 records in 0+0 records out 0 bytes copied, 0.00181408 s, 0.0 kB/s mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl13 cl13' lustre-MDT0001: clear the changelog for cl13 of all records lustre-MDT0001: Deregistered changelog user #13 lustre-MDT0000: clear the changelog for cl13 of all records lustre-MDT0000: Deregistered changelog user #13 PASS 227 (7s) == sanity-hsm test 228: On released file, return extend to FIEMAP. For [cp,tar] --sparse ========================================================== 01:17:21 (1780809441) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_228/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022827 s, 45.9 MB/s Waiting 200s for 'SUCCEED' /mnt/lustre/f228.sanity-hsm: 1 extent found tar: Removing leading `/' from member names mnt/lustre/f228.sanity-hsm PASS 228 (7s) == sanity-hsm test 250: Coordinator max request ========== 01:17:28 (1780809448) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_250/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes copied, 0.00130884 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00123092 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00111551 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00127798 s, 0.0 kB/s Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' 0+0 records in 0+0 records out 0 bytes copied, 0.00154195 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00140797 s, 0.0 kB/s Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Copytool is suspended on oleg406-client.virtnet Waiting 200s for 'STARTED' Updated after 2s: want 'STARTED' got 'STARTED' Copytool is continued on oleg406-client.virtnet PASS 250 (16s) == sanity-hsm test 251: Coordinator request timeout ====== 01:17:44 (1780809464) 0+0 records in 0+0 records out 0 bytes copied, 0.00319582 s, 0.0 kB/s mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_251/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Copytool is suspended on oleg406-client.virtnet Waiting 200s for 'STARTED' Waiting 200s for 'CANCELED' PASS 251 (11s) == sanity-hsm test 252: Timeout'ed running archive of a removed file should be canceled ========================================================== 01:17:55 (1780809475) 0+0 records in 0+0 records out 0 bytes copied, 0.00135813 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_252/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Copytool is suspended on oleg406-client.virtnet Waiting 200s for 'STARTED' Waiting 200s for 'CANCELED' Updated after 2s: want 'CANCELED' got 'CANCELED' Copytool is continued on oleg406-client.virtnet PASS 252 (9s) == sanity-hsm test 253: Check for wrong file size after release ========================================================== 01:18:04 (1780809484) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_253/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 10+0 records in 10+0 records out 10000000 bytes (10 MB, 9.5 MiB) copied, 0.0904067 s, 111 MB/s Waiting 200s for 'SUCCEED' fail_loc=0x807 Cannot send HSM request (use of /mnt/lustre/d253.sanity-hsm/f253.sanity-hsm): Invalid argument could not release file PASS 253 (4s) == sanity-hsm test 254a: Request counters are initialized to zero ========================================================== 01:18:08 (1780809488) PASS 254a (4s) == sanity-hsm test 254b: Request counters are correctly incremented and decremented ========================================================== 01:18:12 (1780809492) Will launch 59 requests of each type Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_254b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Checking archive requests Copytool is suspended on oleg406-client.virtnet Copytool is continued on oleg406-client.virtnet Checking restore requests Copytool is suspended on oleg406-client.virtnet Copytool is continued on oleg406-client.virtnet Checking remove requests Copytool is suspended on oleg406-client.virtnet Copytool is continued on oleg406-client.virtnet PASS 254b (16s) == sanity-hsm test 255: Copytool registration wakes the coordinator up ========================================================== 01:18:28 (1780809508) 0+0 records in 0+0 records out 0 bytes copied, 0.0011688 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_255/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Killing existing copytools on oleg406-client.virtnet with signal TERM copytools stopped in 0s 0+0 records in 0+0 records out 0 bytes copied, 0.00104065 s, 0.0 kB/s 0x200000bd3:0xa9:0x0 mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_255/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' PASS 255 (13s) == sanity-hsm test 260a: Restore request have priority over other requests ========================================================== 01:18:41 (1780809521) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056595 s, 18.5 MB/s 0x200000bd3:0xab:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552938 s, 19.0 MB/s 0x200000bd3:0xac:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496811 s, 21.1 MB/s 0x200000bd3:0xad:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546819 s, 19.2 MB/s 0x200000bd3:0xae:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551426 s, 19.0 MB/s 0x200000bd3:0xaf:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475556 s, 22.0 MB/s 0x200000bd3:0xb0:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470715 s, 22.3 MB/s 0x200000bd3:0xb1:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480218 s, 21.8 MB/s 0x200000bd3:0xb2:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542842 s, 19.3 MB/s 0x200000bd3:0xb3:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056002 s, 18.7 MB/s 0x200000bd3:0xb4:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050425 s, 20.8 MB/s 0x200000bd3:0xb5:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521606 s, 20.1 MB/s 0x200000bd3:0xb6:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569239 s, 18.4 MB/s 0x200000bd3:0xb7:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046532 s, 22.5 MB/s 0x200000bd3:0xb8:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476223 s, 22.0 MB/s 0x200000bd3:0xb9:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045204 s, 23.2 MB/s 0x200000bd3:0xba:0x0 Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Killing existing copytools on oleg406-client.virtnet with signal TERM copytools stopped in 0s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' ARCHIVE ARCHIVE RESTORE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE PASS 260a (15s) == sanity-hsm test 260b: Restore request have priority over other requests ========================================================== 01:18:56 (1780809536) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524151 s, 20.0 MB/s 0x200000bd3:0xbd:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530212 s, 19.8 MB/s 0x200000bd3:0xbe:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561404 s, 18.7 MB/s 0x200000bd3:0xbf:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473415 s, 22.1 MB/s 0x200000bd3:0xc0:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490217 s, 21.4 MB/s 0x200000bd3:0xc1:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478554 s, 21.9 MB/s 0x200000bd3:0xc2:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478837 s, 21.9 MB/s 0x200000bd3:0xc3:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447473 s, 23.4 MB/s 0x200000bd3:0xc4:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047525 s, 22.1 MB/s 0x200000bd3:0xc5:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605658 s, 17.3 MB/s 0x200000bd3:0xc6:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507325 s, 20.7 MB/s 0x200000bd3:0xc7:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488127 s, 21.5 MB/s 0x200000bd3:0xc8:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451739 s, 23.2 MB/s 0x200000bd3:0xc9:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443787 s, 23.6 MB/s 0x200000bd3:0xca:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493135 s, 21.3 MB/s 0x200000bd3:0xcb:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595723 s, 17.6 MB/s 0x200000bd3:0xcc:0x0 Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2 "/mnt/lustre2"' Killing existing copytools on oleg406-client.virtnet with signal TERM copytools stopped in 0s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2 "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' RESTORE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE PASS 260b (16s) == sanity-hsm test 260c: Requests are not reordered on the 'hot' path of the coordinator ========================================================== 01:19:12 (1780809552) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530208 s, 19.8 MB/s 0x200000bd3:0xcf:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547005 s, 19.2 MB/s 0x200000bd3:0xd0:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445195 s, 23.6 MB/s 0x200000bd3:0xd1:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573017 s, 18.3 MB/s 0x200000bd3:0xd2:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530808 s, 19.8 MB/s 0x200000bd3:0xd3:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568776 s, 18.4 MB/s 0x200000bd3:0xd4:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503763 s, 20.8 MB/s 0x200000bd3:0xd5:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536525 s, 19.5 MB/s 0x200000bd3:0xd6:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484966 s, 21.6 MB/s 0x200000bd3:0xd7:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502675 s, 20.9 MB/s 0x200000bd3:0xd8:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454453 s, 23.1 MB/s 0x200000bd3:0xd9:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565218 s, 18.6 MB/s 0x200000bd3:0xda:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597643 s, 17.5 MB/s 0x200000bd3:0xdb:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489338 s, 21.4 MB/s 0x200000bd3:0xdc:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521112 s, 20.1 MB/s 0x200000bd3:0xdd:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477764 s, 21.9 MB/s 0x200000bd3:0xde:0x0 Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2 "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Killing existing copytools on oleg406-client.virtnet with signal TERM copytools stopped in 0s mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2 "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE RESTORE PASS 260c (17s) == sanity-hsm test 261: Report 0 bytes size after HSM release ========================================================== 01:19:29 (1780809569) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_261/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00313749 s, 2.6 MB/s Waiting 200s for 'SUCCEED' /mnt/lustre/d261.sanity-hsm/f261.sanity-hsm: (0x00000009) exists archived, archive_id:2 /mnt/lustre/d261.sanity-hsm/f261.sanity-hsm: (0x0000000d) released exists archived, archive_id:2 PASS 261 (5s) == sanity-hsm test 262: The client should return 1 block for HSM released files ========================================================== 01:19:34 (1780809574) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_262/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00287517 s, 2.8 MB/s Waiting 200s for 'SUCCEED' PASS 262 (4s) == sanity-hsm test 300: On disk coordinator state kept between MDT umount/mount ========================================================== 01:19:38 (1780809578) Stop coordinator and remove coordinator state at mount mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown Failing mds1 on oleg406-server Stopping /mnt/lustre-mds1 (opts:) on oleg406-server 01:19:43 (1780809583) shut down facet: mds1 facet_host: oleg406-server facet_failover_host: oleg406-server Failover mds1 to oleg406-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 01:19:57 (1780809597) targets are mounted 01:19:57 (1780809597) facet_failover done oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Set coordinator start at mount, and start coordinator Failing mds1 on oleg406-server Stopping /mnt/lustre-mds1 (opts:) on oleg406-server 01:20:27 (1780809627) shut down facet: mds1 facet_host: oleg406-server facet_failover_host: oleg406-server Failover mds1 to oleg406-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 01:20:41 (1780809641) targets are mounted 01:20:41 (1780809641) facet_failover done oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 300 (71s) == sanity-hsm test 301: HSM tunnable are persistent ====== 01:20:49 (1780809649) Failing mds1 on oleg406-server Stopping /mnt/lustre-mds1 (opts:) on oleg406-server 01:20:52 (1780809652) shut down facet: mds1 facet_host: oleg406-server facet_failover_host: oleg406-server Failover mds1 to oleg406-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 01:21:06 (1780809666) targets are mounted 01:21:06 (1780809666) facet_failover done oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 301 (26s) == sanity-hsm test 302: HSM tunnable are persistent when CDT is off ========================================================== 01:21:15 (1780809675) mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown Failing mds1 on oleg406-server Stopping /mnt/lustre-mds1 (opts:) on oleg406-server 01:21:19 (1780809679) shut down facet: mds1 facet_host: oleg406-server facet_failover_host: oleg406-server Failover mds1 to oleg406-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 01:21:33 (1780809693) targets are mounted 01:21:33 (1780809693) facet_failover done oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg406-server Stopping /mnt/lustre-mds2 (opts:) on oleg406-server 01:21:40 (1780809700) shut down facet: mds2 facet_host: oleg406-server facet_failover_host: oleg406-server Failover mds2 to oleg406-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg406-server: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0001 01:21:54 (1780809714) targets are mounted 01:21:54 (1780809714) facet_failover done oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 302 (48s) == sanity-hsm test 400: Single request is sent to the right MDT ========================================================== 01:22:03 (1780809723) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_400/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625939 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539608 s, 19.4 MB/s Waiting 200s for 'SUCCEED' archive successful on mdt0 Waiting 200s for 'SUCCEED' archive successful on mdt1 PASS 400 (6s) == sanity-hsm test 401: Compound requests split and sent to their respective MDTs ========================================================== 01:22:09 (1780809729) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_401/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593515 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507024 s, 20.7 MB/s Waiting 200s for 'SUCCEED' archive successful on mdt0 archive successful on mdt1 PASS 401 (4s) == sanity-hsm test 402a: Copytool start fails if all MDTs are inactive ========================================================== 01:22:13 (1780809733) deactivate lustre-MDT0000-mdc-ffff90f185c0d000 on oleg406-client.virtnet deactivate lustre-MDT0001-mdc-ffff90f185c0d000 on oleg406-client.virtnet deactivate lustre-MDT0000-mdc-ffff90f1831cd800 on oleg406-client.virtnet deactivate lustre-MDT0001-mdc-ffff90f1831cd800 on oleg406-client.virtnet Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_402a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Failed to start copytool agt1 on 'oleg406-client.virtnet' pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 uuid not found in agent list on mds1 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 uuid not found in agent list on mds2 activate lustre-MDT0000-mdc-ffff90f185c0d000 on oleg406-client.virtnet activate lustre-MDT0001-mdc-ffff90f185c0d000 on oleg406-client.virtnet activate lustre-MDT0000-mdc-ffff90f1831cd800 on oleg406-client.virtnet activate lustre-MDT0001-mdc-ffff90f1831cd800 on oleg406-client.virtnet PASS 402a (4s) == sanity-hsm test 402b: CDT must retry request upon slow start of CT ========================================================== 01:22:17 (1780809737) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_402b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' fail_loc=0x14d fail_loc=0 Waiting 200s for 'SUCCEED' Updated after 5s: want 'SUCCEED' got 'SUCCEED' PASS 402b (18s) == sanity-hsm test 403: Copytool starts with inactive MDT and register on reconnect ========================================================== 01:22:35 (1780809755) deactivate lustre-MDT0001-mdc-ffff90f185c0d000 on oleg406-client.virtnet deactivate lustre-MDT0001-mdc-ffff90f1831cd800 on oleg406-client.virtnet Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_403/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' oleg406-client.virtnet: executing pgrep --pidfile=/var/run/lhsmtool_posix.pid --list-full hsmtool oleg406-client.virtnet: executing get_client_uuid /mnt/lustre2 found agent uuid=c90ca412-7c0e-430e-8b00-175241015fe3 archive_id=ANY requests=[current:0 ok:0 errors:0] on mds1 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 uuid not found in agent list on mds2 oleg406-client.virtnet: 135623 activate lustre-MDT0001-mdc-ffff90f185c0d000 on oleg406-client.virtnet activate lustre-MDT0001-mdc-ffff90f1831cd800 on oleg406-client.virtnet found agent uuid=c90ca412-7c0e-430e-8b00-175241015fe3 archive_id=ANY requests=[current:0 ok:0 errors:0] on mds1 found agent uuid=c90ca412-7c0e-430e-8b00-175241015fe3 archive_id=ANY requests=[current:0 ok:0 errors:0] on mds2 PASS 403 (6s) == sanity-hsm test 404: Inactive MDT does not block requests for active MDTs ========================================================== 01:22:41 (1780809761) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_404/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060343 s, 17.4 MB/s deactivate lustre-MDT0001-mdc-ffff90f185c0d000 on oleg406-client.virtnet deactivate lustre-MDT0001-mdc-ffff90f1831cd800 on oleg406-client.virtnet Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' archive successful on mdt0 activate lustre-MDT0001-mdc-ffff90f185c0d000 on oleg406-client.virtnet activate lustre-MDT0001-mdc-ffff90f1831cd800 on oleg406-client.virtnet PASS 404 (4s) == sanity-hsm test 405: archive and release under striped directory ========================================================== 01:22:45 (1780809765) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_405/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219859 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018737 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190829 s, 54.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220544 s, 47.5 MB/s Waiting 200s for 'SUCCEED' archive successful on 0x240000bd2:0x1:0x0 archive successful on 0x240000bd2:0x2:0x0 archive successful on 0x240000bd2:0x3:0x0 archive successful on 0x2000032e3:0x6:0x0 PASS 405 (8s) == sanity-hsm test 406: attempting to migrate HSM archived files is safe ========================================================== 01:22:54 (1780809774) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585049 s, 17.9 MB/s old fid 0x2000032e3:0x9:0x0 Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_406/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' lfs migrate: /mnt/lustre/d406.sanity-hsm/f406.sanity-hsm migrate failed: Operation not supported (95) lfs migrate: cb_migrate_mdt_fini: error completing migration of /mnt/lustre/d406.sanity-hsm: Directory not empty (39) Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' new fid 0x240000bd3:0x7:0x0 debug=+trace Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' PASS 406 (8s) == sanity-hsm test 407: Check for double RESTORE records in llog ========================================================== 01:23:01 (1780809781) 0+0 records in 0+0 records out 0 bytes copied, 0.00127304 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_407/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' fail_val=5 fail_loc=0x164 Copytool is suspended on oleg406-client.virtnet mdt.lustre-MDT0000.hsm.actions= lrh=[type=10680000 len=136 idx=6/1] fid=[0x200000bd3:0xe5:0x0] dfid=[0x200000bd3:0xe5:0x0] compound/cookie=0x0/0x6a24ffdc action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/2] fid=[0x200000bd3:0xe9:0x0] dfid=[0x200000bd3:0xe9:0x0] compound/cookie=0x0/0x6a24ffdd action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/3] fid=[0x2000032e2:0x2:0x0] dfid=[0x2000032e2:0x2:0x0] compound/cookie=0x0/0x6a24ffde action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/4] fid=[0x2000032e3:0x3:0x0] dfid=[0x2000032e3:0x3:0x0] compound/cookie=0x0/0x6a24ffdf action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/5] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x6:0x0] compound/cookie=0x0/0x6a24ffe0 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/6] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x7:0x0] compound/cookie=0x0/0x6a24ffe1 action=RESTORE archive#=3 flags=0x4 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/7] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x6a24ffe2 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/8] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0xa:0x0] compound/cookie=0x0/0x6a24ffe3 action=RESTORE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/9] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x6a24ffe4 action=REMOVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/10] fid=[0x2000032e3:0xc:0x0] dfid=[0x2000032e3:0xc:0x0] compound/cookie=0x0/0x6a24ffe5 action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] Failing mds1 on oleg406-server Stopping /mnt/lustre-mds1 (opts:) on oleg406-server 01:23:09 (1780809789) shut down facet: mds1 facet_host: oleg406-server facet_failover_host: oleg406-server Failover mds1 to oleg406-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 01:23:22 (1780809802) targets are mounted 01:23:22 (1780809802) facet_failover done oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 mdt.lustre-MDT0000.hsm.actions= lrh=[type=10680000 len=136 idx=6/3] fid=[0x2000032e2:0x2:0x0] dfid=[0x2000032e2:0x2:0x0] compound/cookie=0x0/0x6a24ffde action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/4] fid=[0x2000032e3:0x3:0x0] dfid=[0x2000032e3:0x3:0x0] compound/cookie=0x0/0x6a24ffdf action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/5] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x6:0x0] compound/cookie=0x0/0x6a24ffe0 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/6] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x7:0x0] compound/cookie=0x0/0x6a24ffe1 action=RESTORE archive#=3 flags=0x4 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/7] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x6a24ffe2 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/8] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0xa:0x0] compound/cookie=0x0/0x6a24ffe3 action=RESTORE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/9] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x6a24ffe4 action=REMOVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/10] fid=[0x2000032e3:0xc:0x0] dfid=[0x2000032e3:0xc:0x0] compound/cookie=0x0/0x6a24ffe5 action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/11] fid=[0x2000032e3:0xc:0x0] dfid=[0x2000032e3:0xc:0x0] compound/cookie=0x0/0x6a24ffe6 action=RESTORE archive#=2 flags=0x4 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[] Copytool is continued on oleg406-client.virtnet d41d8cd98f00b204e9800998ecf8427e /mnt/lustre/d407.sanity-hsm/f407.sanity-hsm d41d8cd98f00b204e9800998ecf8427e /mnt/lustre2/d407.sanity-hsm/f407.sanity-hsm pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 mdt.lustre-MDT0000.hsm.actions= lrh=[type=10680000 len=136 idx=6/3] fid=[0x2000032e2:0x2:0x0] dfid=[0x2000032e2:0x2:0x0] compound/cookie=0x0/0x6a24ffde action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/4] fid=[0x2000032e3:0x3:0x0] dfid=[0x2000032e3:0x3:0x0] compound/cookie=0x0/0x6a24ffdf action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/5] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x6:0x0] compound/cookie=0x0/0x6a24ffe0 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/6] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x7:0x0] compound/cookie=0x0/0x6a24ffe1 action=RESTORE archive#=3 flags=0x4 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/7] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x6a24ffe2 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/8] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0xa:0x0] compound/cookie=0x0/0x6a24ffe3 action=RESTORE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/9] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x6a24ffe4 action=REMOVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/10] fid=[0x2000032e3:0xc:0x0] dfid=[0x2000032e3:0xc:0x0] compound/cookie=0x0/0x6a24ffe5 action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/11] fid=[0x2000032e3:0xc:0x0] dfid=[0x2000032e3:0xd:0x0] compound/cookie=0x0/0x6a24ffe6 action=RESTORE archive#=2 flags=0x4 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] PASS 407 (34s) == sanity-hsm test 408: Verify fiemap on release file ==== 01:23:35 (1780809815) Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_408/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.031527 s, 2.1 MB/s SKIP: sanity-hsm test_408 ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 408 (2s) == sanity-hsm test 409a: Coordinator should not stop when in use ========================================================== 01:23:37 (1780809817) 0+0 records in 0+0 records out 0 bytes copied, 0.00125094 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_409a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' fail_val=5 fail_loc=0x164 cat: /mnt/lustre/d409a.sanity-hsm/f409a.sanity-hsm: No data available mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled Killing existing copytools on oleg406-client.virtnet with signal TERM copytools stopped in 0s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_409a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 409a (14s) == sanity-hsm test 409b: getattr released file with CDT stopped after remount ========================================================== 01:23:51 (1780809831) 0+0 records in 0+0 records out 0 bytes copied, 0.00127763 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_409b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Killing existing copytools on oleg406-client.virtnet with signal TERM copytools stopped in 0s Failing mds1 on oleg406-server Stopping /mnt/lustre-mds1 (opts:) on oleg406-server 01:24:16 (1780809856) shut down facet: mds1 facet_host: oleg406-server facet_failover_host: oleg406-server Failover mds1 to oleg406-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 01:24:30 (1780809870) targets are mounted 01:24:30 (1780809870) facet_failover done oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec File: /mnt/lustre/d409b.sanity-hsm/f409b.sanity-hsm Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115406632648720 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-07 01:23:51.000000000 -0400 Modify: 2026-06-07 01:23:51.000000000 -0400 Change: 2026-06-07 01:23:51.000000000 -0400 Birth: 2026-06-07 01:23:51.000000000 -0400 PASS 409b (68s) == sanity-hsm test 410: lfs data_version -s allows release of force-archived file ========================================================== 01:24:59 (1780809899) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792441 s, 13.2 MB/s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_410/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Cannot send HSM request (use of /mnt/lustre/d410.sanity-hsm/f410.sanity-hsm): Operation not permitted PASS 410 (3s) == sanity-hsm test 411: hsm_ops rbac role ================ 01:25:02 (1780809902) On MGS 192.168.204.106, active = nodemap.active=1 On MGS 192.168.204.106, test_411.trusted_nodemap = nodemap.test_411.trusted_nodemap=1 Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_411/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' /mnt/lustre/d411.sanity-hsm/fileA: (0x00000000) Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' Killing existing copytools on oleg406-client.virtnet with signal TERM copytools stopped in 0s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_411/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' On MGS 192.168.204.106, test_411.rbac = nodemap.test_411.rbac=file_perms,dne_ops,quota_ops,byfid_ops,chlg_ops,fscrypt_admin,server_upcall,ignore_root_prjquota,local_admin,pool_quota_ops,lqa_quota_ops,projid_set,foreign_ops /mnt/lustre/d411.sanity-hsm/fileB: (0x00000000) Cannot send HSM request (use of /mnt/lustre/d411.sanity-hsm/fileB): Permission denied Cannot send HSM request (use of /mnt/lustre/d411.sanity-hsm/fileB): Operation not permitted Cannot send HSM request (use of /mnt/lustre/d411.sanity-hsm/fileB): Permission denied Cannot send HSM request (use of /mnt/lustre/d411.sanity-hsm/fileB): Permission denied lfs hsm_set: change hsm flags for '/mnt/lustre/d411.sanity-hsm/fileB' failed: Permission denied lfs hsm_clear: change hsm flags for '/mnt/lustre/d411.sanity-hsm/fileB' failed: Permission denied On MGS 192.168.204.106, active = nodemap.active=0 oleg406-server: error: nodemap 'test_411' does not exist oleg406-server: nodemap_del: Invalid argument pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 On MGS 192.168.204.106, active = nodemap.active=0 PASS 411 (16s) == sanity-hsm test 500: various LLAPI HSM tests ========== 01:25:18 (1780809918) striped dir -i0 -c2 -H all_char /mnt/lustre/d500.sanity-hsm test1 @1780809937: Register/unregister copytool 2000x to check for leaks ...... pass test2 @1780809937: Re/un-register copytool multiple times without error ....... pass llapi_hsm_test: NULL archive numbers llapi_hsm_test: NULL archive numbers llapi_hsm_test: -1 requested when archive id >= 0 is supported test3 @1780809937: Pass bad parameters to llapi_hsm_copytool_register() ....... pass test4 @1780809937: Bad parameters to llapi_hsm_copytool_unregister() .......... pass test5 @1780809937: Test llapi_hsm_copytool_recv() in non-blocking mode ........ pass test6 @1780809937: Test llapi_hsm_copytool_recv() with bogus parameters ....... pass test7 @1780809937: Test event polling (without actual traffic) ................ pass test50 @1780809937: Test llapi_hsm_state_get()/get_fd() ....................... pass test51 @1780809937: Test llapi_hsm_state_set_fd() ............................. pass test52 @1780809937: Test llapi_hsm_current_action() ........................... pass test100 @1780809938: Simple archive creation with no progress reported. ....... pass test101 @1780809940: Simple archive creation with progress every byte. ........ pass test102 @1780809942: Archive creation with progress every byte, backwards ..... pass test103 @1780809943: Archive creation with a single progress report ........... pass test104 @1780809944: Archive creation with two progress reports ............... pass test105 @1780809945: Archive creation with one bogus progress report .......... pass test106 @1780809946: Archive creation with one empty progress report .......... pass test107 @1780809948: Archive creation with one bogus progress report .......... pass test108 @1780809949: Archive creation with same progress report each time ..... pass test109 @1780809951: Archive creation with one report, with large number ...... pass test110 @1780809952: Archive with 10 progress reports, checking progress ...... pass test111 @1780809953: Archive with 10 reports in reverse, checking progress .... pass test112 @1780809954: Archive with 10 reports, duplicated, check progress ...... pass test113 @1780809955: Archive with 9 reports, with 20% overlapping coverage .... pass PASS 500 (39s) == sanity-hsm test 600: Changelog fields 'u=' and 'nid=' ========================================================== 01:25:57 (1780809957) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d600.sanity-hsm/f600.sanity-hsm] Got NID '192.168.204.6@tcp' lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 PASS 600 (6s) == sanity-hsm test 601: OPEN Changelog entry ============= 01:26:03 (1780809963) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl2 cl2' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0001.4 10OPEN 05:26:05.682354746 2026.06.07 0x1 t=[0x240000bd3:0xa:0x0] j=cat.0 ef=0x17 u=0:0 nid=192.168.204.6@tcp m=r-- p=[0x200000bd2:0x7a:0x0] lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0001: Deregistered changelog user #2 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 PASS 601 (6s) == sanity-hsm test 602: Changelog record CLOSE only if open+write or OPEN recorded ========================================================== 01:26:09 (1780809969) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl3 cl3' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0001: clear the changelog for cl3 of all records lustre-MDT0000.9 11CLOSE 05:26:12.044823414 2026.06.07 0x1 t=[0x2000032e2:0x2d:0x0] j=cat.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0001: clear the changelog for cl3 of all records lustre-MDT0000.10 10OPEN 05:26:12.163497341 2026.06.07 0x242 t=[0x2000032e2:0x2d:0x0] j=bash.0 ef=0x17 u=0:0 nid=192.168.204.6@tcp m=-w- p=[0x2000032e2:0x2a:0x0] lustre-MDT0000.11 13TRUNC 05:26:12.166005364 2026.06.07 0xe t=[0x2000032e2:0x2d:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp p=[0x2000032e2:0x2c:0x0] lustre-MDT0000.12 11CLOSE 05:26:12.173531210 2026.06.07 0x242 t=[0x2000032e2:0x2d:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp lustre-MDT0000.12 11CLOSE 05:26:12.173531210 2026.06.07 0x242 t=[0x2000032e2:0x2d:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp mdd.lustre-MDT0000.changelog_mask=-OPEN mdd.lustre-MDT0001.changelog_mask=-OPEN lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0001: clear the changelog for cl3 of all records f lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0001: clear the changelog for cl3 of all records lustre-MDT0000.13 13TRUNC 05:26:12.912455788 2026.06.07 0xe t=[0x2000032e2:0x2d:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp p=[0x2000032e2:0x2c:0x0] lustre-MDT0000.14 11CLOSE 05:26:12.917589163 2026.06.07 0x242 t=[0x2000032e2:0x2d:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp lustre-MDT0000.14 11CLOSE 05:26:12.917589163 2026.06.07 0x242 t=[0x2000032e2:0x2d:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp lustre-MDT0001: clear the changelog for cl3 of all records lustre-MDT0001: Deregistered changelog user #3 lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0000: Deregistered changelog user #3 PASS 602 (7s) == sanity-hsm test 603: GETXATTR Changelog entry ========= 01:26:16 (1780809976) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl4 cl4' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL lustre-MDT0000: clear the changelog for cl4 of all records lustre-MDT0001: clear the changelog for cl4 of all records getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/d603.sanity-hsm/f603.sanity-hsm user.xattr1="value1" lustre-MDT0001.10 23GXATR 05:26:19.154090077 2026.06.07 0x0 t=[0x240000bd3:0xc:0x0] j=getfattr.0 ef=0x1b u=0:0 nid=192.168.204.6@tcp x=user.xattr1 lustre-MDT0001: clear the changelog for cl4 of all records lustre-MDT0001: Deregistered changelog user #4 lustre-MDT0000: clear the changelog for cl4 of all records lustre-MDT0000: Deregistered changelog user #4 PASS 603 (6s) == sanity-hsm test 604: NOPEN Changelog entry ============ 01:26:22 (1780809982) mdd.lustre-MDT0000.changelog_deniednext=20 mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl5 cl5' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0001: clear the changelog for cl5 of all records running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cat] [/mnt/lustre2/d604.sanity-hsm/f604.sanity-hsm] cat: /mnt/lustre2/d604.sanity-hsm/f604.sanity-hsm: Permission denied lustre-MDT0000.19 24NOPEN 05:26:25.906773910 2026.06.07 0x1 t=[0x2000032e2:0x2f:0x0] j=cat.500 ef=0x17 u=500:500 nid=192.168.204.6@tcp m=r-- p=[0x200000bd2:0x85:0x0] Got NID '192.168.204.6@tcp' lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0001: clear the changelog for cl5 of all records running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cat] [/mnt/lustre2/d604.sanity-hsm/f604.sanity-hsm] cat: /mnt/lustre2/d604.sanity-hsm/f604.sanity-hsm: Permission denied lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0001: clear the changelog for cl5 of all records running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cat] [/mnt/lustre2/d604.sanity-hsm/f604.sanity-hsm] cat: /mnt/lustre2/d604.sanity-hsm/f604.sanity-hsm: Permission denied lustre-MDT0000.20 24NOPEN 05:26:46.246535235 2026.06.07 0x1 t=[0x2000032e2:0x2f:0x0] j=cat.500 ef=0x17 u=500:500 nid=192.168.204.6@tcp m=r-- p=[0x2000032e2:0x2c:0x0] Got NID '192.168.204.6@tcp' lustre-MDT0001: clear the changelog for cl5 of all records lustre-MDT0001: Deregistered changelog user #5 lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0000: Deregistered changelog user #5 mdd.lustre-MDT0000.changelog_deniednext=60 PASS 604 (28s) == sanity-hsm test 605: Test OPEN and CLOSE rate limit in Changelogs ========================================================== 01:26:50 (1780810010) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl6 cl6' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0000.24 10OPEN 05:26:53.017111106 2026.06.07 0x43 t=[0x2000032e2:0x31:0x0] j=bash.0 ef=0x17 u=0:0 nid=192.168.204.6@tcp m=rw- p=[0x200000bd2:0x85:0x0] lustre-MDT0000.24 10OPEN 05:26:53.017111106 2026.06.07 0x43 t=[0x2000032e2:0x31:0x0] j=bash.0 ef=0x17 u=0:0 nid=192.168.204.6@tcp m=rw- p=[0x200000bd2:0x85:0x0] lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0000.25 10OPEN 05:26:53.379852688 2026.06.07 0x1 t=[0x2000032e2:0x31:0x0] j=cat.0 ef=0x17 u=0:0 nid=192.168.204.6@tcp m=r-- lustre-MDT0000.26 11CLOSE 05:26:53.387425491 2026.06.07 0x1 t=[0x2000032e2:0x31:0x0] j=cat.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp lustre-MDT0000.25 10OPEN 05:26:53.379852688 2026.06.07 0x1 t=[0x2000032e2:0x31:0x0] j=cat.0 ef=0x17 u=0:0 nid=192.168.204.6@tcp m=r-- lustre-MDT0000.26 11CLOSE 05:26:53.387425491 2026.06.07 0x1 t=[0x2000032e2:0x31:0x0] j=cat.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0001: clear the changelog for cl6 of all records running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cat] [/mnt/lustre/d605.sanity-hsm/f605.sanity-hsm] lustre-MDT0000.27 10OPEN 05:26:53.805577235 2026.06.07 0x1 t=[0x2000032e2:0x31:0x0] j=cat.500 ef=0x17 u=500:500 nid=192.168.204.6@tcp m=r-- p=[0x200000bd2:0x85:0x0] lustre-MDT0000.28 11CLOSE 05:26:53.809485816 2026.06.07 0x1 t=[0x2000032e2:0x31:0x0] j=cat.500 ef=0x13 u=500:500 nid=192.168.204.6@tcp lustre-MDT0000.27 10OPEN 05:26:53.805577235 2026.06.07 0x1 t=[0x2000032e2:0x31:0x0] j=cat.500 ef=0x17 u=500:500 nid=192.168.204.6@tcp m=r-- p=[0x200000bd2:0x85:0x0] lustre-MDT0000.28 11CLOSE 05:26:53.809485816 2026.06.07 0x1 t=[0x2000032e2:0x31:0x0] j=cat.500 ef=0x13 u=500:500 nid=192.168.204.6@tcp lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0000.29 11CLOSE 05:26:54.024218485 2026.06.07 0x43 t=[0x2000032e2:0x31:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp lustre-MDT0000.29 11CLOSE 05:26:54.024218485 2026.06.07 0x43 t=[0x2000032e2:0x31:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.204.6@tcp lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0001: Deregistered changelog user #6 lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0000: Deregistered changelog user #6 PASS 605 (7s) == sanity-hsm test 606: llog_reader groks changelog fields ========================================================== 01:26:57 (1780810017) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl7 cl7' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d606.sanity-hsm/f606.sanity-hsm] Stopping /mnt/lustre-mds1 (opts:) on oleg406-server rec #2 type=1064553b len=64 offset 8256 Header size : 8192 llh_size : 64 Time : Sun Jun 7 01:13:46 2026 Number of records: 1 cat_idx: 0 last_idx: 2 Flags: 0x2012 Target uuid : ----------------------- #02 (064)fid=[0x1:0x3b:0x0] path=oi.1/0x1:0x3b:0x0 Got UID/GID 500:500 Got NID 192.168.204.6@tcp Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 lustre-MDT0001: clear the changelog for cl7 of all records lustre-MDT0001: Deregistered changelog user #7 lustre-MDT0000: clear the changelog for cl7 of all records lustre-MDT0000: Deregistered changelog user #7 PASS 606 (19s) == sanity-hsm test 607a: release file migrated after archive (non-blocking migrate) ========================================================== 01:27:16 (1780810036) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942085 s, 11.1 MB/s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_607a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' PASS 607a (6s) == sanity-hsm test 607b: Migrate should not change the HSM attribute of dirty files ========================================================== 01:27:22 (1780810042) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883563 s, 11.9 MB/s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_607b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' llite.lustre-ffff90f1831cd800.xattr_cache=0 llite.lustre-ffff90f185c0d000.xattr_cache=0 /mnt/lustre/d607b.sanity-hsm/f607b.sanity-hsm: (0x0000000b) exists dirty archived, archive_id:1 llite.lustre-ffff90f1831cd800.xattr_cache=1 llite.lustre-ffff90f185c0d000.xattr_cache=1 PASS 607b (4s) == sanity-hsm test 607c: 'lfs swap_layouts' should set dirty flag on HSM file ========================================================== 01:27:26 (1780810046) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816041 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845653 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949355 s, 11.0 MB/s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_607c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' lfs swap_layouts: error: cannot swap layout between '/mnt/lustre/d607c.sanity-hsm/f607c.sanity-hsm-1' and '/mnt/lustre/d607c.sanity-hsm/f607c.sanity-hsm-3': Operation not permitted (1) PASS 607c (5s) == sanity-hsm test 607d: Migrate should not set HSM dirty flag for an empty file ========================================================== 01:27:31 (1780810051) 0+0 records in 0+0 records out 0 bytes copied, 0.00133027 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_607d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' PASS 607d (5s) == sanity-hsm test 607e: release file migrated after archive (blocking migrate) ========================================================== 01:27:37 (1780810057) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790435 s, 13.3 MB/s Starting copytool 'agt1' on 'oleg406-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_607e/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' PASS 607e (5s) == sanity-hsm test complete, duration 2168 sec =========== 01:27:41 (1780810061) === sanity-hsm: start cleanup 01:27:42 (1780810062) === Stopping clients: oleg406-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg406-client.virtnet /mnt/lustre2 opts: === sanity-hsm: finish cleanup 01:27:51 (1780810071) ===