-----============= acceptance-small: sanity-hsm ============----- Tue Aug 5 10:51:03 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg305-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-hsm.*ex': No such file or directory excepting tests: === sanity-hsm: start setup 10:51:12 (1754405472) === Starting client oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 Started clients oleg305-client.virtnet: 192.168.203.105@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg305-client.virtnet: executing check_config_client /mnt/lustre oleg305-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg305-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8df9504f8800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8df950d18000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8df9504f8800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8df950d18000.idle_timeout=debug disable quota as required oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-hsm: finish setup 10:51:27 (1754405487) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8192] nobody:x:65534: Killing existing copytools on oleg305-client.virtnet pkill: pidfile not valid Try `pkill --help' for more information. Set HSM on and start Set sanity-hsm HSM policy == sanity-hsm test 1A: lfs hsm flags root/non-root access ========================================================== 10:51:51 (1754405511) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_set] [--norelease] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_clear] [--norelease] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_set] [--exists] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] lfs hsm_set: change hsm flags for '/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm' failed: Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1A.sanity-hsm/f1A.sanity-hsm] PASS 1A (3s) == sanity-hsm test 1a: mmap & cat a HSM released file ==== 10:51:54 (1754405514) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641675 s, 16.3 MB/s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_1a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Verifying released state: PASS 1a (6s) == sanity-hsm test 1b: Archive, Release and Restore composite file ========================================================== 10:52:00 (1754405520) DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[0, 1048576] DEBUG: prev component: NULL DEBUG: Component starts at 0, checking if new mirror DEBUG: Setting first_comp = true (new mirror) DEBUG: Setting prev = NULL (new mirror) DEBUG: About to check adjacency, prev=NULL DEBUG: Checking start <= end: start=0, end=1048576 DEBUG: Component validation passed, continuing DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[1048576, 67108864] DEBUG: prev component: found DEBUG: prev component flags=0x0, extent=[0, 1048576] DEBUG: About to check adjacency, prev=exists DEBUG: Checking adjacency: prev=[0, 1048576], curr=[1048576, 67108864] DEBUG: prev flags=0x0, curr flags=0x0 DEBUG: Checking start <= end: start=1048576, end=67108864 DEBUG: Component validation passed, continuing DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[67108864, 18446744073709551615] DEBUG: prev component: found DEBUG: prev component flags=0x0, extent=[1048576, 67108864] DEBUG: About to check adjacency, prev=exists DEBUG: Checking adjacency: prev=[1048576, 67108864], curr=[67108864, 18446744073709551615] DEBUG: prev flags=0x0, curr flags=0x0 DEBUG: Checking start <= end: start=67108864, end=18446744073709551615 DEBUG: Component validation passed, continuing 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330238 s, 31.8 MB/s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_1b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' archive /mnt/lustre/d1b.sanity-hsm/f1b.sanity-hsm Waiting 200s for 'SUCCEED' release /mnt/lustre/d1b.sanity-hsm/f1b.sanity-hsm verify released state: pass restore /mnt/lustre/d1b.sanity-hsm/f1b.sanity-hsm Waiting 200s for 'SUCCEED' verify restored state: pass PASS 1b (7s) == sanity-hsm test 1c: Check setting archive-id in lfs hsm_set ========================================================== 10:52:07 (1754405527) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d1c.sanity-hsm/f1c.sanity-hsm] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1c.sanity-hsm/f1c.sanity-hsm] verifying archive number is 32 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1c.sanity-hsm/f1c.sanity-hsm] verifying archive number is still 32 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1c.sanity-hsm/f1c.sanity-hsm] verifying archive number is 33 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d1c.sanity-hsm/f1c.sanity-hsm] verifying archive number is 16 PASS 1c (3s) == sanity-hsm test 1d: Archive, Release and Restore DoM file ========================================================== 10:52:10 (1754405530) DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[0, 1048576] DEBUG: prev component: NULL DEBUG: Component starts at 0, checking if new mirror DEBUG: Setting first_comp = true (new mirror) DEBUG: Setting prev = NULL (new mirror) DEBUG: About to check adjacency, prev=NULL DEBUG: Checking start <= end: start=0, end=1048576 DEBUG: Component validation passed, continuing DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[1048576, 18446744073709551615] DEBUG: prev component: found DEBUG: prev component flags=0x0, extent=[0, 1048576] DEBUG: About to check adjacency, prev=exists DEBUG: Checking adjacency: prev=[0, 1048576], curr=[1048576, 18446744073709551615] DEBUG: prev flags=0x0, curr flags=0x0 DEBUG: Checking start <= end: start=1048576, end=18446744073709551615 DEBUG: Component validation passed, continuing 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027474 s, 38.2 MB/s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_1d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' archive /mnt/lustre/d1d.sanity-hsm/f1d.sanity-hsm release /mnt/lustre/d1d.sanity-hsm/f1d.sanity-hsm verify released state: pass restore /mnt/lustre/d1d.sanity-hsm/f1d.sanity-hsm Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' verify restored state: pass PASS 1d (5s) == sanity-hsm test 1e: Archive, Release and Restore SEL file ========================================================== 10:52:15 (1754405535) DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[0, 67108864] DEBUG: prev component: NULL DEBUG: Component starts at 0, checking if new mirror DEBUG: Setting first_comp = true (new mirror) DEBUG: Setting prev = NULL (new mirror) DEBUG: About to check adjacency, prev=NULL DEBUG: Checking start <= end: start=0, end=67108864 DEBUG: Component validation passed, continuing DEBUG: llapi_layout_sanity_cb: component flags=0x40, extent=[67108864, 1073741824] DEBUG: prev component: found DEBUG: prev component flags=0x0, extent=[0, 67108864] DEBUG: About to check adjacency, prev=exists DEBUG: Checking adjacency: prev=[0, 67108864], curr=[67108864, 1073741824] DEBUG: prev flags=0x0, curr flags=0x40 DEBUG: Checking start <= end: start=67108864, end=1073741824 DEBUG: Component validation passed, continuing DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[1073741824, 1073741824] DEBUG: prev component: found DEBUG: prev component flags=0x40, extent=[67108864, 1073741824] DEBUG: About to check adjacency, prev=exists DEBUG: Checking adjacency: prev=[67108864, 1073741824], curr=[1073741824, 1073741824] DEBUG: prev flags=0x40, curr flags=0x0 DEBUG: Checking start <= end: start=1073741824, end=1073741824 DEBUG: Component validation passed, continuing DEBUG: llapi_layout_sanity_cb: component flags=0x40, extent=[1073741824, 10737418240] DEBUG: prev component: found DEBUG: prev component flags=0x0, extent=[1073741824, 1073741824] DEBUG: About to check adjacency, prev=exists DEBUG: Checking adjacency: prev=[1073741824, 1073741824], curr=[1073741824, 10737418240] DEBUG: prev flags=0x0, curr flags=0x40 DEBUG: Checking start <= end: start=1073741824, end=10737418240 DEBUG: Component validation passed, continuing DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[10737418240, 10737418240] DEBUG: prev component: found DEBUG: prev component flags=0x40, extent=[1073741824, 10737418240] DEBUG: About to check adjacency, prev=exists DEBUG: Checking adjacency: prev=[1073741824, 10737418240], curr=[10737418240, 10737418240] DEBUG: prev flags=0x40, curr flags=0x0 DEBUG: Checking start <= end: start=10737418240, end=10737418240 DEBUG: Component validation passed, continuing DEBUG: llapi_layout_sanity_cb: component flags=0x40, extent=[10737418240, 18446744073709551615] DEBUG: prev component: found DEBUG: prev component flags=0x0, extent=[10737418240, 10737418240] DEBUG: About to check adjacency, prev=exists DEBUG: Checking adjacency: prev=[10737418240, 10737418240], curr=[10737418240, 18446744073709551615] DEBUG: prev flags=0x0, curr flags=0x40 DEBUG: Checking start <= end: start=10737418240, end=18446744073709551615 DEBUG: Component validation passed, continuing 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318586 s, 32.9 MB/s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_1e/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' archive /mnt/lustre/d1e.sanity-hsm/f1e.sanity-hsm release /mnt/lustre/d1e.sanity-hsm/f1e.sanity-hsm verify released state: pass restore /mnt/lustre/d1e.sanity-hsm/f1e.sanity-hsm Waiting 200s for 'SUCCEED' verify restored state: pass PASS 1e (6s) == sanity-hsm test 1f: DoM file release after restore ==== 10:52:21 (1754405541) DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[0, 524288] DEBUG: prev component: NULL DEBUG: Component starts at 0, checking if new mirror DEBUG: Setting first_comp = true (new mirror) DEBUG: Setting prev = NULL (new mirror) DEBUG: About to check adjacency, prev=NULL DEBUG: Checking start <= end: start=0, end=524288 DEBUG: Component validation passed, continuing DEBUG: llapi_layout_sanity_cb: component flags=0x0, extent=[524288, 18446744073709551615] DEBUG: prev component: found DEBUG: prev component flags=0x0, extent=[0, 524288] DEBUG: About to check adjacency, prev=exists DEBUG: Checking adjacency: prev=[0, 524288], curr=[524288, 18446744073709551615] DEBUG: prev flags=0x0, curr flags=0x0 DEBUG: Checking start <= end: start=524288, end=18446744073709551615 DEBUG: Component validation passed, continuing 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382671 s, 27.4 MB/s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_1f/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' archive /mnt/lustre/d1f.sanity-hsm/f1f.sanity-hsm Waiting 200s for 'SUCCEED' release /mnt/lustre/d1f.sanity-hsm/f1f.sanity-hsm verify released state: pass restore /mnt/lustre/d1f.sanity-hsm/f1f.sanity-hsm verify restored state: pass release again /mnt/lustre/d1f.sanity-hsm/f1f.sanity-hsm /mnt/lustre/d1f.sanity-hsm/f1f.sanity-hsm: (0x0000000d) released exists archived, archive_id:1 verify released state: pass PASS 1f (5s) == sanity-hsm test 2: Check file dirtyness when doing setattr ========================================================== 10:52:26 (1754405546) 0+0 records in 0+0 records out 0 bytes copied, 0.00259974 s, 0.0 kB/s 0x200000403:0x8:0x0 PASS 2 (4s) == sanity-hsm test 3: Check file dirtyness when opening for write ========================================================== 10:52:30 (1754405550) Succeed in opening file "/mnt/lustre/d3.sanity-hsm/f3.sanity-hsm"(flags=O_WRONLY) 3+0 records in 3+0 records out 3+0 records in 3+0 records out PASS 3 (3s) == sanity-hsm test 4: Useless cancel must not be registered ========================================================== 10:52:33 (1754405553) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465228 s, 22.5 MB/s PASS 4 (2s) == sanity-hsm test 8: Test default archive number ======== 10:52:35 (1754405555) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_8/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' PASS 8 (3s) == sanity-hsm test 9A: Use of explicit archive number, with dedicated copytool ========================================================== 10:52:38 (1754405558) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_9A/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 3 "/mnt/lustre2"' oleg305-client.virtnet: executing pgrep --pidfile=/var/run/lhsmtool_posix.pid --list-full hsmtool oleg305-client.virtnet: executing get_client_uuid /mnt/lustre2 found agent uuid=4a43a402-6c57-4091-9ce2-335332730d3a archive_id=3 requests=[current:0 ok:0 errors:0] on mds1 found agent uuid=4a43a402-6c57-4091-9ce2-335332730d3a archive_id=3 requests=[current:0 ok:0 errors:0] on mds2 Waiting 200s for 'SUCCEED' PASS 9A (13s) == sanity-hsm test 9a: Multiple remote agents ============ 10:52:51 (1754405571) SKIP: sanity-hsm test_9a Need 3 or more clients, have 1 SKIP 9a (3s) == sanity-hsm test 10a: Archive a file =================== 10:52:54 (1754405574) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_10a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh: line 938: warning: command substitution: ignored null byte in input Verifying content Verifying hsm state Verifying archive number is 2 PASS 10a (10s) == sanity-hsm test 10b: Archive of non dirty file must work without doing request ========================================================== 10:53:04 (1754405584) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_10b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' PASS 10b (8s) == sanity-hsm test 10c: Check forbidden archive ========== 10:53:12 (1754405592) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_10c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Cannot send HSM request (use of /mnt/lustre/d10c.sanity-hsm/f10c.sanity-hsm): Operation not permitted PASS 10c (6s) == sanity-hsm test 10d: Archive a file on the default archive id ========================================================== 10:53:18 (1754405598) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_10d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' PASS 10d (8s) == sanity-hsm test 11a: Import a file ==================== 10:53:26 (1754405606) lhsmtool_posix: 1754405608.461527 [22039]: action=1 src=d11a.sanity-hsm/f11a.sanity-hsm dst=/mnt/lustre/d11a.sanity-hsm/f11a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1754405608.468513 [22039]: importing '/mnt/lustre/d11a.sanity-hsm/f11a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_11a//d11a.sanity-hsm/f11a.sanity-hsm' lhsmtool_posix: 1754405608.520705 [22039]: imported '/mnt/lustre/d11a.sanity-hsm/f11a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_11a//0424/0x200000402:0x26:0x0'=='/tmp/arc1/sanity-hsm.test_11a//d11a.sanity-hsm/f11a.sanity-hsm' lhsmtool_posix: 1754405608.520786 [22039]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Verifying released state: Verifying imported size 158=158 Verifying released pattern: raid0,released Verifying new fid 0x200000402:0x26:0x0 in archive PASS 11a (7s) == sanity-hsm test 11b: Import a deleted file using its FID ========================================================== 10:53:33 (1754405613) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_11b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' lhsmtool_posix: 1754405617.950530 [22738]: action=1 src=0x200000403:0x11:0x0 dst=/mnt/lustre/d11b.sanity-hsm/f11b.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1754405617.963300 [22738]: Resolving [0x200000403:0x11:0x0] to /tmp/arc1/sanity-hsm.test_11b//0412/0x200000403:0x11:0x0 lhsmtool_posix: 1754405617.963385 [22738]: importing '/mnt/lustre/d11b.sanity-hsm/f11b.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_11b//0412/0x200000403:0x11:0x0' lhsmtool_posix: 1754405618.007668 [22738]: imported '/mnt/lustre/d11b.sanity-hsm/f11b.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_11b//042a/0x200000402:0x28:0x0'=='/tmp/arc1/sanity-hsm.test_11b//0412/0x200000403:0x11:0x0' lhsmtool_posix: 1754405618.007723 [22738]: process finished, errs: 0 major, 0 minor, rc=0 (Success) /mnt/lustre/d11b.sanity-hsm/f11b.sanity-hsm: OK PASS 11b (8s) == sanity-hsm test 11c: Import a file to a directory with a pool ========================================================== 10:53:41 (1754405621) Creating new pool test_11c oleg305-server: Pool lustre.test_11c created Adding targets to pool oleg305-server: OST lustre-OST0001_UUID added to pool lustre.test_11c lhsmtool_posix: 1754405628.905134 [23453]: action=1 src=d11c.sanity-hsm/f11c.sanity-hsm dst=/mnt/lustre/d11c.sanity-hsm/f11c.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1754405628.915381 [23453]: importing '/mnt/lustre/d11c.sanity-hsm/f11c.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_11c//d11c.sanity-hsm/f11c.sanity-hsm' lhsmtool_posix: 1754405628.955939 [23453]: imported '/mnt/lustre/d11c.sanity-hsm/f11c.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_11c//0428/0x200000402:0x2a:0x0'=='/tmp/arc1/sanity-hsm.test_11c//d11c.sanity-hsm/f11c.sanity-hsm' lhsmtool_posix: 1754405628.955998 [23453]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Destroy the created pools: test_11c lustre.test_11c oleg305-server: OST lustre-OST0001_UUID removed from pool lustre.test_11c oleg305-server: Pool lustre.test_11c destroyed PASS 11c (17s) == sanity-hsm test 12a: Restore an imported file explicitly ========================================================== 10:53:58 (1754405638) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1754405639.141557 [24190]: action=1 src=d12a.sanity-hsm/f12a.sanity-hsm dst=/mnt/lustre/d12a.sanity-hsm/f12a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1754405639.146773 [24190]: importing '/mnt/lustre/d12a.sanity-hsm/f12a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12a//d12a.sanity-hsm/f12a.sanity-hsm' lhsmtool_posix: 1754405639.172200 [24190]: imported '/mnt/lustre/d12a.sanity-hsm/f12a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12a//042f/0x200000402:0x2d:0x0'=='/tmp/arc1/sanity-hsm.test_12a//d12a.sanity-hsm/f12a.sanity-hsm' lhsmtool_posix: 1754405639.172333 [24190]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Verifying released state: Waiting 200s for 'SUCCEED' Verifying file state: PASS 12a (5s) == sanity-hsm test 12b: Restore an imported file implicitly ========================================================== 10:54:03 (1754405643) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1754405644.550114 [24905]: action=1 src=d12b.sanity-hsm/f12b.sanity-hsm dst=/mnt/lustre/d12b.sanity-hsm/f12b.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1754405644.554524 [24905]: importing '/mnt/lustre/d12b.sanity-hsm/f12b.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12b//d12b.sanity-hsm/f12b.sanity-hsm' lhsmtool_posix: 1754405644.588187 [24905]: imported '/mnt/lustre/d12b.sanity-hsm/f12b.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12b//042d/0x200000402:0x2f:0x0'=='/tmp/arc1/sanity-hsm.test_12b//d12b.sanity-hsm/f12b.sanity-hsm' lhsmtool_posix: 1754405644.588247 [24905]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Verifying released state: Verifying file state after restore: PASS 12b (4s) == sanity-hsm test 12c: Restore a file with stripe of 2 == 10:54:07 (1754405647) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.0740614 s, 70.8 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' /mnt/lustre/d12c.sanity-hsm/f12c.sanity-hsm: OK PASS 12c (6s) == sanity-hsm test 12d: Restore of a non archived, non released file must work ========================================================== 10:54:13 (1754405653) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' PASS 12d (7s) == sanity-hsm test 12e: Check forbidden restore ========== 10:54:20 (1754405660) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12e/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' /mnt/lustre/d12e.sanity-hsm/f12e.sanity-hsm: (0x0000000b) exists dirty archived, archive_id:1 Cannot send HSM request (use of /mnt/lustre/d12e.sanity-hsm/f12e.sanity-hsm): Operation not permitted PASS 12e (5s) == sanity-hsm test 12f: Restore a released file explicitly ========================================================== 10:54:25 (1754405665) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12f/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Verifying file state: PASS 12f (6s) == sanity-hsm test 12g: Restore a released file implicitly ========================================================== 10:54:32 (1754405672) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12g/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' PASS 12g (7s) == sanity-hsm test 12h: Restore a released file implicitly from a second node ========================================================== 10:54:38 (1754405678) SKIP: sanity-hsm test_12h Need 2 or more clients, have 1 SKIP 12h (2s) == sanity-hsm test 12m: Archive/release/implicit restore ========================================================== 10:54:40 (1754405680) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12m/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' PASS 12m (6s) == sanity-hsm test 12n: Import/implicit restore/release == 10:54:46 (1754405686) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12n/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1754405687.840869 [29765]: action=1 src=d12n.sanity-hsm/f12n.sanity-hsm dst=/mnt/lustre/d12n.sanity-hsm/f12n.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1754405687.848492 [29765]: importing '/mnt/lustre/d12n.sanity-hsm/f12n.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12n//d12n.sanity-hsm/f12n.sanity-hsm' lhsmtool_posix: 1754405687.881972 [29765]: imported '/mnt/lustre/d12n.sanity-hsm/f12n.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12n//043f/0x200000402:0x3d:0x0'=='/tmp/arc1/sanity-hsm.test_12n//d12n.sanity-hsm/f12n.sanity-hsm' lhsmtool_posix: 1754405687.882036 [29765]: process finished, errs: 0 major, 0 minor, rc=0 (Success) PASS 12n (4s) == sanity-hsm test 12o: Layout-swap failure during Restore leaves file released ========================================================== 10:54:50 (1754405690) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12o/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' fail_loc=0x152 mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0001.hsm.policy=+NRA diff: /mnt/lustre/d12o.sanity-hsm/f12o.sanity-hsm: No data available mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA fail_loc=0 mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' PASS 12o (26s) == sanity-hsm test 12p: implicit restore of a file on copytool mount point ========================================================== 10:55:16 (1754405716) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12p/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' PASS 12p (8s) == sanity-hsm test 12q: file attributes are refreshed after restore ========================================================== 10:55:24 (1754405724) Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre3 Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12q/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre3"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956842 s, 11.0 MB/s Waiting 200s for 'SUCCEED' Killing existing copytools on oleg305-client.virtnet copytools stopped in 0s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12q/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre3"' Stopping client oleg305-client.virtnet /mnt/lustre3 (opts:) PASS 12q (13s) == sanity-hsm test 12r: lseek restores released file ===== 10:55:37 (1754405737) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12r/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' PASS 12r (6s) == sanity-hsm test 12s: race between restore requests ==== 10:55:43 (1754405743) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12s/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' fail_loc=0x8000018b PASS 12s (13s) == sanity-hsm test 12t: Multiple parallel reads for a HSM imported file ========================================================== 10:55:56 (1754405756) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12t/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' lhsmtool_posix: 1754405759.816884 [33959]: action=1 src=0x200000403:0x2a:0x0 dst=/mnt/lustre/d12t.sanity-hsm/f12t.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1754405759.821448 [33959]: Resolving [0x200000403:0x2a:0x0] to /tmp/arc1/sanity-hsm.test_12t//0429/0x200000403:0x2a:0x0 lhsmtool_posix: 1754405759.821534 [33959]: importing '/mnt/lustre/d12t.sanity-hsm/f12t.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12t//0429/0x200000403:0x2a:0x0' lhsmtool_posix: 1754405759.857202 [33959]: imported '/mnt/lustre/d12t.sanity-hsm/f12t.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_12t//0449/0x200000402:0x4b:0x0'=='/tmp/arc1/sanity-hsm.test_12t//0429/0x200000403:0x2a:0x0' lhsmtool_posix: 1754405759.857274 [33959]: process finished, errs: 0 major, 0 minor, rc=0 (Success) PASS 12t (7s) == sanity-hsm test 12u: Multiple reads on multiple HSM imported files in parallel ========================================================== 10:56:03 (1754405763) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_12u/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1754405770.240315 [34826]: action=1 src=0x200000403:0x2c:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.0 mount_point=/mnt/lustre2 lhsmtool_posix: 1754405770.249893 [34826]: Resolving [0x200000403:0x2c:0x0] to /tmp/arc1/sanity-hsm.test_12u//042f/0x200000403:0x2c:0x0 lhsmtool_posix: 1754405770.249966 [34826]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.0' from '/tmp/arc1/sanity-hsm.test_12u//042f/0x200000403:0x2c:0x0' lhsmtool_posix: 1754405770.276295 [34826]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.0' from '/tmp/arc1/sanity-hsm.test_12u//044f/0x200000402:0x4d:0x0'=='/tmp/arc1/sanity-hsm.test_12u//042f/0x200000403:0x2c:0x0' lhsmtool_posix: 1754405770.276351 [34826]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1754405770.424772 [34855]: action=1 src=0x200000403:0x2d:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.1 mount_point=/mnt/lustre2 lhsmtool_posix: 1754405770.431106 [34855]: Resolving [0x200000403:0x2d:0x0] to /tmp/arc1/sanity-hsm.test_12u//042e/0x200000403:0x2d:0x0 lhsmtool_posix: 1754405770.431190 [34855]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.1' from '/tmp/arc1/sanity-hsm.test_12u//042e/0x200000403:0x2d:0x0' lhsmtool_posix: 1754405770.460856 [34855]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.1' from '/tmp/arc1/sanity-hsm.test_12u//044c/0x200000402:0x4e:0x0'=='/tmp/arc1/sanity-hsm.test_12u//042e/0x200000403:0x2d:0x0' lhsmtool_posix: 1754405770.460913 [34855]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1754405770.592406 [34884]: action=1 src=0x200000403:0x2e:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.2 mount_point=/mnt/lustre2 lhsmtool_posix: 1754405770.600299 [34884]: Resolving [0x200000403:0x2e:0x0] to /tmp/arc1/sanity-hsm.test_12u//042d/0x200000403:0x2e:0x0 lhsmtool_posix: 1754405770.600390 [34884]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.2' from '/tmp/arc1/sanity-hsm.test_12u//042d/0x200000403:0x2e:0x0' lhsmtool_posix: 1754405770.632520 [34884]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.2' from '/tmp/arc1/sanity-hsm.test_12u//044d/0x200000402:0x4f:0x0'=='/tmp/arc1/sanity-hsm.test_12u//042d/0x200000403:0x2e:0x0' lhsmtool_posix: 1754405770.632574 [34884]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1754405770.763391 [34914]: action=1 src=0x200000403:0x2f:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.3 mount_point=/mnt/lustre2 lhsmtool_posix: 1754405770.770302 [34914]: Resolving [0x200000403:0x2f:0x0] to /tmp/arc1/sanity-hsm.test_12u//042c/0x200000403:0x2f:0x0 lhsmtool_posix: 1754405770.770394 [34914]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.3' from '/tmp/arc1/sanity-hsm.test_12u//042c/0x200000403:0x2f:0x0' lhsmtool_posix: 1754405770.813312 [34914]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.3' from '/tmp/arc1/sanity-hsm.test_12u//0452/0x200000402:0x50:0x0'=='/tmp/arc1/sanity-hsm.test_12u//042c/0x200000403:0x2f:0x0' lhsmtool_posix: 1754405770.813371 [34914]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1754405770.955910 [34943]: action=1 src=0x200000403:0x30:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.4 mount_point=/mnt/lustre2 lhsmtool_posix: 1754405770.963217 [34943]: Resolving [0x200000403:0x30:0x0] to /tmp/arc1/sanity-hsm.test_12u//0433/0x200000403:0x30:0x0 lhsmtool_posix: 1754405770.963315 [34943]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.4' from '/tmp/arc1/sanity-hsm.test_12u//0433/0x200000403:0x30:0x0' lhsmtool_posix: 1754405771.002202 [34943]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.4' from '/tmp/arc1/sanity-hsm.test_12u//0453/0x200000402:0x51:0x0'=='/tmp/arc1/sanity-hsm.test_12u//0433/0x200000403:0x30:0x0' lhsmtool_posix: 1754405771.003906 [34943]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1754405771.158832 [34972]: action=1 src=0x200000403:0x31:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.5 mount_point=/mnt/lustre2 lhsmtool_posix: 1754405771.165266 [34972]: Resolving [0x200000403:0x31:0x0] to /tmp/arc1/sanity-hsm.test_12u//0432/0x200000403:0x31:0x0 lhsmtool_posix: 1754405771.165355 [34972]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.5' from '/tmp/arc1/sanity-hsm.test_12u//0432/0x200000403:0x31:0x0' lhsmtool_posix: 1754405771.198150 [34972]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.5' from '/tmp/arc1/sanity-hsm.test_12u//0450/0x200000402:0x52:0x0'=='/tmp/arc1/sanity-hsm.test_12u//0432/0x200000403:0x31:0x0' lhsmtool_posix: 1754405771.198207 [34972]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1754405771.336540 [35001]: action=1 src=0x200000403:0x32:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.6 mount_point=/mnt/lustre2 lhsmtool_posix: 1754405771.343181 [35001]: Resolving [0x200000403:0x32:0x0] to /tmp/arc1/sanity-hsm.test_12u//0431/0x200000403:0x32:0x0 lhsmtool_posix: 1754405771.343277 [35001]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.6' from '/tmp/arc1/sanity-hsm.test_12u//0431/0x200000403:0x32:0x0' lhsmtool_posix: 1754405771.365105 [35001]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.6' from '/tmp/arc1/sanity-hsm.test_12u//0451/0x200000402:0x53:0x0'=='/tmp/arc1/sanity-hsm.test_12u//0431/0x200000403:0x32:0x0' lhsmtool_posix: 1754405771.365162 [35001]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1754405771.527106 [35030]: action=1 src=0x200000403:0x33:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.7 mount_point=/mnt/lustre2 lhsmtool_posix: 1754405771.530825 [35030]: Resolving [0x200000403:0x33:0x0] to /tmp/arc1/sanity-hsm.test_12u//0430/0x200000403:0x33:0x0 lhsmtool_posix: 1754405771.530906 [35030]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.7' from '/tmp/arc1/sanity-hsm.test_12u//0430/0x200000403:0x33:0x0' lhsmtool_posix: 1754405771.565995 [35030]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.7' from '/tmp/arc1/sanity-hsm.test_12u//0456/0x200000402:0x54:0x0'=='/tmp/arc1/sanity-hsm.test_12u//0430/0x200000403:0x33:0x0' lhsmtool_posix: 1754405771.566053 [35030]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1754405771.708602 [35059]: action=1 src=0x200000403:0x34:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.8 mount_point=/mnt/lustre2 lhsmtool_posix: 1754405771.715399 [35059]: Resolving [0x200000403:0x34:0x0] to /tmp/arc1/sanity-hsm.test_12u//0437/0x200000403:0x34:0x0 lhsmtool_posix: 1754405771.715506 [35059]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.8' from '/tmp/arc1/sanity-hsm.test_12u//0437/0x200000403:0x34:0x0' lhsmtool_posix: 1754405771.743278 [35059]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.8' from '/tmp/arc1/sanity-hsm.test_12u//0457/0x200000402:0x55:0x0'=='/tmp/arc1/sanity-hsm.test_12u//0437/0x200000403:0x34:0x0' lhsmtool_posix: 1754405771.743352 [35059]: process finished, errs: 0 major, 0 minor, rc=0 (Success) lhsmtool_posix: 1754405771.881117 [35088]: action=1 src=0x200000403:0x35:0x0 dst=/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.9 mount_point=/mnt/lustre2 lhsmtool_posix: 1754405771.885566 [35088]: Resolving [0x200000403:0x35:0x0] to /tmp/arc1/sanity-hsm.test_12u//0436/0x200000403:0x35:0x0 lhsmtool_posix: 1754405771.885661 [35088]: importing '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.9' from '/tmp/arc1/sanity-hsm.test_12u//0436/0x200000403:0x35:0x0' lhsmtool_posix: 1754405771.913395 [35088]: imported '/mnt/lustre/d12u.sanity-hsm/f12u.sanity-hsm.9' from '/tmp/arc1/sanity-hsm.test_12u//0454/0x200000402:0x56:0x0'=='/tmp/arc1/sanity-hsm.test_12u//0436/0x200000403:0x35:0x0' lhsmtool_posix: 1754405771.913451 [35088]: process finished, errs: 0 major, 0 minor, rc=0 (Success) PASS 12u (12s) == sanity-hsm test 13: Recursively import and restore a directory ========================================================== 10:56:15 (1754405775) lhsmtool_posix: 1754405779.468156 [36836]: action=1 src=subdir dst=/mnt/lustre/d13.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1754405779.529442 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.10' lhsmtool_posix: 1754405779.561304 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.10' from '/tmp/arc1/sanity-hsm.test_13//045b/0x200000402:0x59:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.10' lhsmtool_posix: 1754405779.561511 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.9' lhsmtool_posix: 1754405779.589965 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.9' from '/tmp/arc1/sanity-hsm.test_13//0458/0x200000402:0x5a:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.9' lhsmtool_posix: 1754405779.590178 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.8' lhsmtool_posix: 1754405779.617406 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.8' from '/tmp/arc1/sanity-hsm.test_13//0459/0x200000402:0x5b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.8' lhsmtool_posix: 1754405779.617670 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.7' lhsmtool_posix: 1754405779.643889 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.7' from '/tmp/arc1/sanity-hsm.test_13//045e/0x200000402:0x5c:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.7' lhsmtool_posix: 1754405779.644086 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.6' lhsmtool_posix: 1754405779.665986 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.6' from '/tmp/arc1/sanity-hsm.test_13//045f/0x200000402:0x5d:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.6' lhsmtool_posix: 1754405779.666159 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.5' lhsmtool_posix: 1754405779.692658 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.5' from '/tmp/arc1/sanity-hsm.test_13//045c/0x200000402:0x5e:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.5' lhsmtool_posix: 1754405779.693419 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.4' lhsmtool_posix: 1754405779.717872 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.4' from '/tmp/arc1/sanity-hsm.test_13//045d/0x200000402:0x5f:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.4' lhsmtool_posix: 1754405779.718030 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.3' lhsmtool_posix: 1754405779.739403 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.3' from '/tmp/arc1/sanity-hsm.test_13//0462/0x200000402:0x60:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.3' lhsmtool_posix: 1754405779.739836 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.2' lhsmtool_posix: 1754405779.764226 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.2' from '/tmp/arc1/sanity-hsm.test_13//0463/0x200000402:0x61:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.2' lhsmtool_posix: 1754405779.764414 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.1' lhsmtool_posix: 1754405779.785922 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.10/file.1' from '/tmp/arc1/sanity-hsm.test_13//0460/0x200000402:0x62:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.10/file.1' lhsmtool_posix: 1754405779.788949 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.10' lhsmtool_posix: 1754405779.817207 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.10' from '/tmp/arc1/sanity-hsm.test_13//0408/0x240000402:0xa:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.10' lhsmtool_posix: 1754405779.817396 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.9' lhsmtool_posix: 1754405779.841696 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.9' from '/tmp/arc1/sanity-hsm.test_13//0409/0x240000402:0xb:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.9' lhsmtool_posix: 1754405779.842014 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.8' lhsmtool_posix: 1754405779.864547 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.8' from '/tmp/arc1/sanity-hsm.test_13//040e/0x240000402:0xc:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.8' lhsmtool_posix: 1754405779.864731 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.7' lhsmtool_posix: 1754405779.889842 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.7' from '/tmp/arc1/sanity-hsm.test_13//040f/0x240000402:0xd:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.7' lhsmtool_posix: 1754405779.890090 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.6' lhsmtool_posix: 1754405779.908563 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.6' from '/tmp/arc1/sanity-hsm.test_13//040c/0x240000402:0xe:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.6' lhsmtool_posix: 1754405779.908813 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.5' lhsmtool_posix: 1754405779.931740 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.5' from '/tmp/arc1/sanity-hsm.test_13//040d/0x240000402:0xf:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.5' lhsmtool_posix: 1754405779.932046 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.4' lhsmtool_posix: 1754405779.952621 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.4' from '/tmp/arc1/sanity-hsm.test_13//0412/0x240000402:0x10:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.4' lhsmtool_posix: 1754405779.952848 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.3' lhsmtool_posix: 1754405779.972277 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.3' from '/tmp/arc1/sanity-hsm.test_13//0413/0x240000402:0x11:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.3' lhsmtool_posix: 1754405779.972457 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.2' lhsmtool_posix: 1754405779.994881 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.2' from '/tmp/arc1/sanity-hsm.test_13//0410/0x240000402:0x12:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.2' lhsmtool_posix: 1754405779.995119 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.1' lhsmtool_posix: 1754405780.017390 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.9/file.1' from '/tmp/arc1/sanity-hsm.test_13//0411/0x240000402:0x13:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.9/file.1' lhsmtool_posix: 1754405780.026798 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.10' lhsmtool_posix: 1754405780.056395 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.10' from '/tmp/arc1/sanity-hsm.test_13//0466/0x200000402:0x64:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.10' lhsmtool_posix: 1754405780.056572 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.9' lhsmtool_posix: 1754405780.074500 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.9' from '/tmp/arc1/sanity-hsm.test_13//0467/0x200000402:0x65:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.9' lhsmtool_posix: 1754405780.075037 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.8' lhsmtool_posix: 1754405780.102969 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.8' from '/tmp/arc1/sanity-hsm.test_13//0464/0x200000402:0x66:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.8' lhsmtool_posix: 1754405780.103305 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.7' lhsmtool_posix: 1754405780.126126 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.7' from '/tmp/arc1/sanity-hsm.test_13//0465/0x200000402:0x67:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.7' lhsmtool_posix: 1754405780.126300 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.6' lhsmtool_posix: 1754405780.158696 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.6' from '/tmp/arc1/sanity-hsm.test_13//046a/0x200000402:0x68:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.6' lhsmtool_posix: 1754405780.160355 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.5' lhsmtool_posix: 1754405780.193166 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.5' from '/tmp/arc1/sanity-hsm.test_13//046b/0x200000402:0x69:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.5' lhsmtool_posix: 1754405780.193413 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.4' lhsmtool_posix: 1754405780.218937 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.4' from '/tmp/arc1/sanity-hsm.test_13//0468/0x200000402:0x6a:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.4' lhsmtool_posix: 1754405780.219188 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.3' lhsmtool_posix: 1754405780.255821 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.3' from '/tmp/arc1/sanity-hsm.test_13//0469/0x200000402:0x6b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.3' lhsmtool_posix: 1754405780.256057 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.2' lhsmtool_posix: 1754405780.282431 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.2' from '/tmp/arc1/sanity-hsm.test_13//046e/0x200000402:0x6c:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.2' lhsmtool_posix: 1754405780.282673 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.1' lhsmtool_posix: 1754405780.307077 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.8/file.1' from '/tmp/arc1/sanity-hsm.test_13//046f/0x200000402:0x6d:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.8/file.1' lhsmtool_posix: 1754405780.310079 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.10' lhsmtool_posix: 1754405780.334472 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.10' from '/tmp/arc1/sanity-hsm.test_13//0417/0x240000402:0x15:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.10' lhsmtool_posix: 1754405780.334677 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.9' lhsmtool_posix: 1754405780.362010 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.9' from '/tmp/arc1/sanity-hsm.test_13//0414/0x240000402:0x16:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.9' lhsmtool_posix: 1754405780.362226 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.8' lhsmtool_posix: 1754405780.383899 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.8' from '/tmp/arc1/sanity-hsm.test_13//0415/0x240000402:0x17:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.8' lhsmtool_posix: 1754405780.384108 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.7' lhsmtool_posix: 1754405780.405678 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.7' from '/tmp/arc1/sanity-hsm.test_13//041a/0x240000402:0x18:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.7' lhsmtool_posix: 1754405780.405915 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.6' lhsmtool_posix: 1754405780.429295 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.6' from '/tmp/arc1/sanity-hsm.test_13//041b/0x240000402:0x19:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.6' lhsmtool_posix: 1754405780.429519 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.5' lhsmtool_posix: 1754405780.451295 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.5' from '/tmp/arc1/sanity-hsm.test_13//0418/0x240000402:0x1a:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.5' lhsmtool_posix: 1754405780.451838 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.4' lhsmtool_posix: 1754405780.475520 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.4' from '/tmp/arc1/sanity-hsm.test_13//0419/0x240000402:0x1b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.4' lhsmtool_posix: 1754405780.475738 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.3' lhsmtool_posix: 1754405780.511351 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.3' from '/tmp/arc1/sanity-hsm.test_13//041e/0x240000402:0x1c:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.3' lhsmtool_posix: 1754405780.511592 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.2' lhsmtool_posix: 1754405780.538658 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.2' from '/tmp/arc1/sanity-hsm.test_13//041f/0x240000402:0x1d:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.2' lhsmtool_posix: 1754405780.538984 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.1' lhsmtool_posix: 1754405780.560306 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.7/file.1' from '/tmp/arc1/sanity-hsm.test_13//041c/0x240000402:0x1e:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.7/file.1' lhsmtool_posix: 1754405780.569788 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.10' lhsmtool_posix: 1754405780.599502 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.10' from '/tmp/arc1/sanity-hsm.test_13//046d/0x200000402:0x6f:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.10' lhsmtool_posix: 1754405780.599695 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.9' lhsmtool_posix: 1754405780.621183 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.9' from '/tmp/arc1/sanity-hsm.test_13//0472/0x200000402:0x70:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.9' lhsmtool_posix: 1754405780.621389 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.8' lhsmtool_posix: 1754405780.638886 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.8' from '/tmp/arc1/sanity-hsm.test_13//0473/0x200000402:0x71:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.8' lhsmtool_posix: 1754405780.639068 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.7' lhsmtool_posix: 1754405780.660275 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.7' from '/tmp/arc1/sanity-hsm.test_13//0470/0x200000402:0x72:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.7' lhsmtool_posix: 1754405780.660482 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.6' lhsmtool_posix: 1754405780.686825 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.6' from '/tmp/arc1/sanity-hsm.test_13//0471/0x200000402:0x73:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.6' lhsmtool_posix: 1754405780.687094 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.5' lhsmtool_posix: 1754405780.709210 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.5' from '/tmp/arc1/sanity-hsm.test_13//0476/0x200000402:0x74:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.5' lhsmtool_posix: 1754405780.709455 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.4' lhsmtool_posix: 1754405780.730391 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.4' from '/tmp/arc1/sanity-hsm.test_13//0477/0x200000402:0x75:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.4' lhsmtool_posix: 1754405780.730583 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.3' lhsmtool_posix: 1754405780.751805 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.3' from '/tmp/arc1/sanity-hsm.test_13//0474/0x200000402:0x76:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.3' lhsmtool_posix: 1754405780.751993 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.2' lhsmtool_posix: 1754405780.783633 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.2' from '/tmp/arc1/sanity-hsm.test_13//0475/0x200000402:0x77:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.2' lhsmtool_posix: 1754405780.783921 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.1' lhsmtool_posix: 1754405780.807112 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.6/file.1' from '/tmp/arc1/sanity-hsm.test_13//047a/0x200000402:0x78:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.6/file.1' lhsmtool_posix: 1754405780.810010 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.10' lhsmtool_posix: 1754405780.829842 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.10' from '/tmp/arc1/sanity-hsm.test_13//0422/0x240000402:0x20:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.10' lhsmtool_posix: 1754405780.830051 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.9' lhsmtool_posix: 1754405780.852851 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.9' from '/tmp/arc1/sanity-hsm.test_13//0423/0x240000402:0x21:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.9' lhsmtool_posix: 1754405780.853031 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.8' lhsmtool_posix: 1754405780.883436 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.8' from '/tmp/arc1/sanity-hsm.test_13//0420/0x240000402:0x22:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.8' lhsmtool_posix: 1754405780.883905 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.7' lhsmtool_posix: 1754405780.919927 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.7' from '/tmp/arc1/sanity-hsm.test_13//0421/0x240000402:0x23:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.7' lhsmtool_posix: 1754405780.920163 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.6' lhsmtool_posix: 1754405780.949414 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.6' from '/tmp/arc1/sanity-hsm.test_13//0426/0x240000402:0x24:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.6' lhsmtool_posix: 1754405780.949615 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.5' lhsmtool_posix: 1754405780.973711 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.5' from '/tmp/arc1/sanity-hsm.test_13//0427/0x240000402:0x25:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.5' lhsmtool_posix: 1754405780.973951 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.4' lhsmtool_posix: 1754405780.993358 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.4' from '/tmp/arc1/sanity-hsm.test_13//0424/0x240000402:0x26:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.4' lhsmtool_posix: 1754405780.993611 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.3' lhsmtool_posix: 1754405781.015215 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.3' from '/tmp/arc1/sanity-hsm.test_13//0425/0x240000402:0x27:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.3' lhsmtool_posix: 1754405781.015472 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.2' lhsmtool_posix: 1754405781.049479 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.2' from '/tmp/arc1/sanity-hsm.test_13//042a/0x240000402:0x28:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.2' lhsmtool_posix: 1754405781.049792 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.1' lhsmtool_posix: 1754405781.074388 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.5/file.1' from '/tmp/arc1/sanity-hsm.test_13//042b/0x240000402:0x29:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.5/file.1' lhsmtool_posix: 1754405781.083011 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.10' lhsmtool_posix: 1754405781.126112 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.10' from '/tmp/arc1/sanity-hsm.test_13//0478/0x200000402:0x7a:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.10' lhsmtool_posix: 1754405781.126308 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.9' lhsmtool_posix: 1754405781.149215 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.9' from '/tmp/arc1/sanity-hsm.test_13//0479/0x200000402:0x7b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.9' lhsmtool_posix: 1754405781.149411 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.8' lhsmtool_posix: 1754405781.173890 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.8' from '/tmp/arc1/sanity-hsm.test_13//047e/0x200000402:0x7c:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.8' lhsmtool_posix: 1754405781.174073 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.7' lhsmtool_posix: 1754405781.194258 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.7' from '/tmp/arc1/sanity-hsm.test_13//047f/0x200000402:0x7d:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.7' lhsmtool_posix: 1754405781.194445 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.6' lhsmtool_posix: 1754405781.218214 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.6' from '/tmp/arc1/sanity-hsm.test_13//047c/0x200000402:0x7e:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.6' lhsmtool_posix: 1754405781.218401 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.5' lhsmtool_posix: 1754405781.242034 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.5' from '/tmp/arc1/sanity-hsm.test_13//047d/0x200000402:0x7f:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.5' lhsmtool_posix: 1754405781.242229 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.4' lhsmtool_posix: 1754405781.261450 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.4' from '/tmp/arc1/sanity-hsm.test_13//0482/0x200000402:0x80:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.4' lhsmtool_posix: 1754405781.261647 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.3' lhsmtool_posix: 1754405781.292408 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.3' from '/tmp/arc1/sanity-hsm.test_13//0483/0x200000402:0x81:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.3' lhsmtool_posix: 1754405781.292684 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.2' lhsmtool_posix: 1754405781.321010 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.2' from '/tmp/arc1/sanity-hsm.test_13//0480/0x200000402:0x82:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.2' lhsmtool_posix: 1754405781.321246 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.1' lhsmtool_posix: 1754405781.357146 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.4/file.1' from '/tmp/arc1/sanity-hsm.test_13//0481/0x200000402:0x83:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.4/file.1' lhsmtool_posix: 1754405781.363092 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.10' lhsmtool_posix: 1754405781.400084 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.10' from '/tmp/arc1/sanity-hsm.test_13//0429/0x240000402:0x2b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.10' lhsmtool_posix: 1754405781.400301 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.9' lhsmtool_posix: 1754405781.445809 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.9' from '/tmp/arc1/sanity-hsm.test_13//042e/0x240000402:0x2c:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.9' lhsmtool_posix: 1754405781.448964 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.8' lhsmtool_posix: 1754405781.483120 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.8' from '/tmp/arc1/sanity-hsm.test_13//042f/0x240000402:0x2d:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.8' lhsmtool_posix: 1754405781.483357 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.7' lhsmtool_posix: 1754405781.519894 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.7' from '/tmp/arc1/sanity-hsm.test_13//042c/0x240000402:0x2e:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.7' lhsmtool_posix: 1754405781.520102 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.6' lhsmtool_posix: 1754405781.554436 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.6' from '/tmp/arc1/sanity-hsm.test_13//042d/0x240000402:0x2f:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.6' lhsmtool_posix: 1754405781.554648 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.5' lhsmtool_posix: 1754405781.597253 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.5' from '/tmp/arc1/sanity-hsm.test_13//0432/0x240000402:0x30:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.5' lhsmtool_posix: 1754405781.597421 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.4' lhsmtool_posix: 1754405781.629961 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.4' from '/tmp/arc1/sanity-hsm.test_13//0433/0x240000402:0x31:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.4' lhsmtool_posix: 1754405781.630595 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.3' lhsmtool_posix: 1754405781.656003 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.3' from '/tmp/arc1/sanity-hsm.test_13//0430/0x240000402:0x32:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.3' lhsmtool_posix: 1754405781.656193 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.2' lhsmtool_posix: 1754405781.683824 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.2' from '/tmp/arc1/sanity-hsm.test_13//0431/0x240000402:0x33:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.2' lhsmtool_posix: 1754405781.684017 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.1' lhsmtool_posix: 1754405781.710282 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.3/file.1' from '/tmp/arc1/sanity-hsm.test_13//0436/0x240000402:0x34:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.3/file.1' lhsmtool_posix: 1754405781.720588 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.10' lhsmtool_posix: 1754405781.759939 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.10' from '/tmp/arc1/sanity-hsm.test_13//0487/0x200000402:0x85:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.10' lhsmtool_posix: 1754405781.760149 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.9' lhsmtool_posix: 1754405781.785703 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.9' from '/tmp/arc1/sanity-hsm.test_13//0484/0x200000402:0x86:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.9' lhsmtool_posix: 1754405781.785978 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.8' lhsmtool_posix: 1754405781.807150 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.8' from '/tmp/arc1/sanity-hsm.test_13//0485/0x200000402:0x87:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.8' lhsmtool_posix: 1754405781.807355 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.7' lhsmtool_posix: 1754405781.836603 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.7' from '/tmp/arc1/sanity-hsm.test_13//048a/0x200000402:0x88:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.7' lhsmtool_posix: 1754405781.836861 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.6' lhsmtool_posix: 1754405781.856901 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.6' from '/tmp/arc1/sanity-hsm.test_13//048b/0x200000402:0x89:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.6' lhsmtool_posix: 1754405781.857067 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.5' lhsmtool_posix: 1754405781.883504 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.5' from '/tmp/arc1/sanity-hsm.test_13//0488/0x200000402:0x8a:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.5' lhsmtool_posix: 1754405781.883697 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.4' lhsmtool_posix: 1754405781.909563 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.4' from '/tmp/arc1/sanity-hsm.test_13//0489/0x200000402:0x8b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.4' lhsmtool_posix: 1754405781.910248 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.3' lhsmtool_posix: 1754405781.936175 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.3' from '/tmp/arc1/sanity-hsm.test_13//048e/0x200000402:0x8c:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.3' lhsmtool_posix: 1754405781.937037 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.2' lhsmtool_posix: 1754405781.963021 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.2' from '/tmp/arc1/sanity-hsm.test_13//048f/0x200000402:0x8d:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.2' lhsmtool_posix: 1754405781.963210 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.1' lhsmtool_posix: 1754405781.988728 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.2/file.1' from '/tmp/arc1/sanity-hsm.test_13//048c/0x200000402:0x8e:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.2/file.1' lhsmtool_posix: 1754405781.993952 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.10' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.10' lhsmtool_posix: 1754405782.026291 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.10' from '/tmp/arc1/sanity-hsm.test_13//0434/0x240000402:0x36:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.10' lhsmtool_posix: 1754405782.026472 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.9' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.9' lhsmtool_posix: 1754405782.059308 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.9' from '/tmp/arc1/sanity-hsm.test_13//0435/0x240000402:0x37:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.9' lhsmtool_posix: 1754405782.059502 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.8' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.8' lhsmtool_posix: 1754405782.091818 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.8' from '/tmp/arc1/sanity-hsm.test_13//043a/0x240000402:0x38:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.8' lhsmtool_posix: 1754405782.092263 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.7' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.7' lhsmtool_posix: 1754405782.114430 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.7' from '/tmp/arc1/sanity-hsm.test_13//043b/0x240000402:0x39:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.7' lhsmtool_posix: 1754405782.114813 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.6' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.6' lhsmtool_posix: 1754405782.141140 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.6' from '/tmp/arc1/sanity-hsm.test_13//0438/0x240000402:0x3a:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.6' lhsmtool_posix: 1754405782.141386 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.5' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.5' lhsmtool_posix: 1754405782.161296 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.5' from '/tmp/arc1/sanity-hsm.test_13//0439/0x240000402:0x3b:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.5' lhsmtool_posix: 1754405782.161486 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.4' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.4' lhsmtool_posix: 1754405782.193077 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.4' from '/tmp/arc1/sanity-hsm.test_13//043e/0x240000402:0x3c:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.4' lhsmtool_posix: 1754405782.193315 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.3' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.3' lhsmtool_posix: 1754405782.224232 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.3' from '/tmp/arc1/sanity-hsm.test_13//043f/0x240000402:0x3d:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.3' lhsmtool_posix: 1754405782.224424 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.2' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.2' lhsmtool_posix: 1754405782.246955 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.2' from '/tmp/arc1/sanity-hsm.test_13//043c/0x240000402:0x3e:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.2' lhsmtool_posix: 1754405782.247167 [36836]: importing '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.1' from '/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.1' lhsmtool_posix: 1754405782.271886 [36836]: imported '/mnt/lustre/d13.sanity-hsm/subdir/dir.1/file.1' from '/tmp/arc1/sanity-hsm.test_13//043d/0x240000402:0x3f:0x0'=='/tmp/arc1/sanity-hsm.test_13//subdir/dir.1/file.1' lhsmtool_posix: 1754405782.271982 [36836]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_13/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' PASS 13 (15s) == sanity-hsm test 14: Rebind archived file to a new fid ========================================================== 10:56:30 (1754405790) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_14/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607015 s, 17.3 MB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00553848 s, 0.0 kB/s rebind 0x200000403:0x73:0x0 to 0x200000403:0x74:0x0 lhsmtool_posix: 1754405792.411356 [37783]: action=2 src=0x200000403:0x73:0x0 dst=0x200000403:0x74:0x0 mount_point=/mnt/lustre2 lhsmtool_posix: 1754405792.412381 [37783]: rebind [0x200000403:0x73:0x0] to [0x200000403:0x74:0x0] lhsmtool_posix: 1754405792.412539 [37783]: process finished, errs: 0 major, 0 minor, rc=0 (Success) PASS 14 (5s) == sanity-hsm test 15: Rebind a list of files ============ 10:56:35 (1754405795) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_15/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487484 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707185 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058623 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637415 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689814 s, 15.2 MB/s Waiting 300s for '' pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 0+0 records in 0+0 records out 0 bytes copied, 0.00665185 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00274901 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00379177 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00843994 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00534647 s, 0.0 kB/s rebind list of files lhsmtool_posix: 1754405800.248661 [38574]: action=2 src=/tmp/tmp.8192 dst=(null) mount_point=/mnt/lustre2 lhsmtool_posix: 1754405800.249721 [38574]: rebind [0x200000403:0x76:0x0] to [0x200000403:0x7b:0x0] lhsmtool_posix: 1754405800.249902 [38574]: rebind [0x200000403:0x77:0x0] to [0x200000403:0x7c:0x0] lhsmtool_posix: 1754405800.249974 [38574]: rebind [0x200000403:0x78:0x0] to [0x200000403:0x7d:0x0] lhsmtool_posix: 1754405800.250024 [38574]: rebind [0x200000403:0x79:0x0] to [0x200000403:0x7e:0x0] lhsmtool_posix: 1754405800.250070 [38574]: rebind [0x200000403:0x7a:0x0] to [0x200000403:0x7f:0x0] lhsmtool_posix: 1754405800.250127 [38574]: 5 lines read from '/tmp/tmp.8192', 5 rebind successful lhsmtool_posix: 1754405800.250143 [38574]: process finished, errs: 0 major, 0 minor, rc=0 (Success) PASS 15 (13s) == sanity-hsm test 16: Test CT bandwith control option === 10:56:48 (1754405808) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_16/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.0369882 s, 567 MB/s Waiting 200s for 'SUCCEED' Waiting 190s for 'SUCCEED' Waiting 180s for 'SUCCEED' Updated after 21s: want 'SUCCEED' got 'SUCCEED' PASS 16 (25s) == sanity-hsm test 20: Release is not permitted ========== 10:57:13 (1754405833) 0+0 records in 0+0 records out 0 bytes copied, 0.00397014 s, 0.0 kB/s 0x200000403:0x87:0x0 Cannot send HSM request (use of /mnt/lustre/d20.sanity-hsm/f20.sanity-hsm): Operation not permitted Cannot send HSM request (use of /mnt/lustre/d20.sanity-hsm/f20.sanity-hsm): Operation not permitted Cannot send HSM request (use of /mnt/lustre/d20.sanity-hsm/f20.sanity-hsm): Operation not permitted Cannot send HSM request (use of /mnt/lustre/d20.sanity-hsm/f20.sanity-hsm): Operation not permitted PASS 20 (3s) == sanity-hsm test 21: Simple release tests ============== 10:57:16 (1754405836) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_21/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927933 s, 11.3 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' PASS 21 (7s) == sanity-hsm test 22: Could not swap a release file ===== 10:57:23 (1754405843) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_22/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642602 s, 16.3 MB/s Waiting 200s for 'SUCCEED' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751948 s, 13.9 MB/s 0x200000403:0x8b:0x0 lfs swap_layouts: error: cannot swap layout between '/mnt/lustre/d22.sanity-hsm/test_swap' and '/mnt/lustre/d22.sanity-hsm/test_release': Operation not supported (95) PASS 22 (5s) == sanity-hsm test 23: Release does not change a/mtime (utime) ========================================================== 10:57:28 (1754405848) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_23/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667754 s, 15.7 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' PASS 23 (6s) == sanity-hsm test 24a: Archive, release, and restore does not change a/mtime (i/o) ========================================================== 10:57:34 (1754405854) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701495 s, 14.9 MB/s Waiting 200s for 'SUCCEED' Killing existing copytools on oleg305-client.virtnet copytools stopped in 0s 192.168.203.105@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg305-client.virtnet /mnt/lustre (opts:) Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre PASS 24a (8s) == sanity-hsm test 24b: root can archive, release, and restore user files ========================================================== 10:57:43 (1754405863) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733988 s, 14.3 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_state] [/mnt/lustre/d24b.sanity-hsm/f24b.sanity-hsm] /mnt/lustre/d24b.sanity-hsm/f24b.sanity-hsm: (0x00000009) exists archived, archive_id:1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [md5sum] [/mnt/lustre/d24b.sanity-hsm/f24b.sanity-hsm] PASS 24b (7s) == sanity-hsm test 24c: check that user,group,other request masks work ========================================================== 10:57:50 (1754405870) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668836 s, 15.7 MB/s 0x200000403:0x94:0x0 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_archive] [/mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm] Cannot send HSM request (use of /mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm): Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_archive] [/mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594699 s, 17.6 MB/s 0x200000403:0x95:0x0 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_archive] [/mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm] Cannot send HSM request (use of /mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm): Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_archive] [/mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491949 s, 21.3 MB/s 0x200000403:0x96:0x0 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_archive] [/mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm] Cannot send HSM request (use of /mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm): Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [hsm_archive] [/mnt/lustre/d24c.sanity-hsm/f24c.sanity-hsm] PASS 24c (10s) == sanity-hsm test 24d: check that read-only mounts are respected ========================================================== 10:58:00 (1754405880) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570639 s, 18.4 MB/s 0x200000403:0x97:0x0 /mnt/lustre/d24d.sanity-hsm/f24d.sanity-hsm lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 81 0x51 0x2c0000401 Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre3 Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre3"' Cannot send HSM request (use of /mnt/lustre2/d24d.sanity-hsm/f24d.sanity-hsm): Read-only file system Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29471 s, 3.6 MB/s Cannot send HSM request (use of /mnt/lustre2/d24d.sanity-hsm/f24d.sanity-hsm): Read-only file system Stopping client oleg305-client.virtnet /mnt/lustre3 (opts:) PASS 24d (9s) == sanity-hsm test 24e: tar succeeds on HSM released files ========================================================== 10:58:09 (1754405889) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24e/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669344 s, 15.7 MB/s /mnt/lustre/d24e.sanity-hsm/f24e.sanity-hsm: (0x0000000d) released exists archived, archive_id:1 tar: Removing leading `/' from member names PASS 24e (5s) == sanity-hsm test 24f: root can archive, release, and restore tar files ========================================================== 10:58:15 (1754405895) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24f/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 54fb6627dbaa37721048e4549db3224d /mnt/lustre/d24f.sanity-hsm/f24f.sanity-hsm f24f.sanity-hsm f24f.sanity-hsm Sum0 = 54fb6627dbaa37721048e4549db3224d /mnt/lustre/d24f.sanity-hsm/f24f.sanity-hsm, sum1 = 54fb6627dbaa37721048e4549db3224d /mnt/lustre/d24f.sanity-hsm/f24f.sanity-hsm PASS 24f (6s) == sanity-hsm test 24g: write by non-owner still sets dirty ========================================================== 10:58:20 (1754405900) RUNAS = 'runas -u 500 -g 500' Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_24g/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Please listen carefully as our options have changed. Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' running as uid/gid/euid/egid 500/500/500/500, groups: 500 [tee] [/mnt/lustre/d24g.sanity-hsm/f24g.sanity-hsm] To be electrocuted by your telephone, press #. PASS 24g (6s) == sanity-hsm test 25a: Restore lost file (HS_LOST flag) from import ========================================================== 10:58:27 (1754405907) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_25a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1754405908.384841 [47631]: action=1 src=d25a.sanity-hsm/f25a.sanity-hsm dst=/mnt/lustre/d25a.sanity-hsm/f25a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1754405908.395437 [47631]: importing '/mnt/lustre/d25a.sanity-hsm/f25a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_25a//d25a.sanity-hsm/f25a.sanity-hsm' lhsmtool_posix: 1754405908.438385 [47631]: imported '/mnt/lustre/d25a.sanity-hsm/f25a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_25a//0406/0x240000404:0x2:0x0'=='/tmp/arc1/sanity-hsm.test_25a//d25a.sanity-hsm/f25a.sanity-hsm' lhsmtool_posix: 1754405908.438447 [47631]: process finished, errs: 0 major, 0 minor, rc=0 (Success) md5sum: /mnt/lustre/d25a.sanity-hsm/f25a.sanity-hsm: No data available PASS 25a (4s) == sanity-hsm test 25b: Restore lost file (HS_LOST flag) after release ========================================================== 10:58:31 (1754405911) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_25b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' md5sum: /mnt/lustre/d25b.sanity-hsm/f25b.sanity-hsm: No data available PASS 25b (6s) == sanity-hsm test 26A: Remove the archive of a valid file ========================================================== 10:58:37 (1754405917) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26A/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes copied, 0.00689773 s, 0.0 kB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' PASS 26A (7s) == sanity-hsm test 26a: Remove Archive On Last Unlink (RAoLU) policy ========================================================== 10:58:44 (1754405924) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' Updated after 8s: want 'SUCCEED' got 'SUCCEED' PASS 26a (26s) == sanity-hsm test 26b: RAoLU policy when CDT off ======== 10:59:10 (1754405950) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled Killing existing copytools on oleg305-client.virtnet copytools stopped in 0s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' PASS 26b (14s) == sanity-hsm test 26c: RAoLU effective when file closed ========================================================== 10:59:24 (1754405964) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' multiop /mnt/lustre/d26c.sanity-hsm/f26c.sanity-hsm vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8192 Waiting 200s for 'SUCCEED' Updated after 9s: want 'SUCCEED' got 'SUCCEED' Waiting 200s for 'SUCCEED' Updated after 10s: want 'SUCCEED' got 'SUCCEED' PASS 26c (32s) == sanity-hsm test 26d: RAoLU when Client eviction ======= 10:59:56 (1754405996) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616887 s, 17.0 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' multiop /mnt/lustre/d26d.sanity-hsm/f26d.sanity-hsm vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8192 Waiting 200s for 'SUCCEED' Updated after 7s: want 'SUCCEED' got 'SUCCEED' PASS 26d (21s) == sanity-hsm test 26e: RAoLU with a non-started coordinator ========================================================== 11:00:17 (1754406017) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26e/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779079 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619039 s, 16.9 MB/s Killing existing copytools on oleg305-client.virtnet copytools stopped in 0s Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 11:00:43 (1754406043) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 11:00:59 (1754406059) targets are mounted 11:00:59 (1754406059) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Check cookie from RAoLU request (last: 1754405654, remove: 1754405655) mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_26e/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' PASS 26e (78s) == sanity-hsm test 27a: Remove the archive of an imported file (Operation not permitted) ========================================================== 11:01:35 (1754406095) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_27a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 39+0 records in 39+0 records out 40894464 bytes (41 MB, 39 MiB) copied, 0.484676 s, 84.4 MB/s lhsmtool_posix: 1754406097.238017 [54829]: action=1 src=d27a.sanity-hsm/f27a.sanity-hsm dst=/mnt/lustre/d27a.sanity-hsm/f27a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1754406097.240836 [54829]: importing '/mnt/lustre/d27a.sanity-hsm/f27a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_27a//d27a.sanity-hsm/f27a.sanity-hsm' lhsmtool_posix: 1754406097.259324 [54829]: imported '/mnt/lustre/d27a.sanity-hsm/f27a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_27a//0400/0x240000404:0x4:0x0'=='/tmp/arc1/sanity-hsm.test_27a//d27a.sanity-hsm/f27a.sanity-hsm' lhsmtool_posix: 1754406097.259374 [54829]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Cannot send HSM request (use of /mnt/lustre/d27a.sanity-hsm/f27a.sanity-hsm): Operation not permitted PASS 27a (4s) == sanity-hsm test 27b: Remove the archive of a relased file (Operation not permitted) ========================================================== 11:01:39 (1754406099) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_27b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes copied, 0.00232922 s, 0.0 kB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Cannot send HSM request (use of /mnt/lustre/d27b.sanity-hsm/f27b.sanity-hsm): Operation not permitted PASS 27b (4s) == sanity-hsm test 28: Concurrent archive/file remove ==== 11:01:43 (1754406103) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_28/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes copied, 0.00246605 s, 0.0 kB/s Waiting 200s for 'SUCCEED' mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 28 (9s) == sanity-hsm test 29a: Tests --mntpath and --archive options ========================================================== 11:01:52 (1754406112) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_29a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 7 "/mnt/lustre"' Cannot send HSM request (use of /mnt/lustre): Invalid argument hsm: '/qwerty/uyt' is not a valid FID PASS 29a (3s) == sanity-hsm test 29b: Archive/delete/remove by FID from the archive. ========================================================== 11:01:55 (1754406115) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_29b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553553 s, 18.9 MB/s Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' PASS 29b (6s) == sanity-hsm test 29c: Archive/delete/remove by FID, using a file list. ========================================================== 11:02:01 (1754406121) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_29c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426693 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045858 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494405 s, 21.2 MB/s Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' PASS 29c (8s) == sanity-hsm test 29d: hsm_remove by FID with archive_id 0 for unlinked file cause ========================================================== 11:02:09 (1754406129) SKIP: sanity-hsm test_29d Need 3 or more clients, have 1 SKIP 29d (1s) == sanity-hsm test 30a: Restore at exec (import case) ==== 11:02:10 (1754406130) SKIP: sanity-hsm test_30a Need 2 or more clients, have 1 SKIP 30a (1s) == sanity-hsm test 30b: Restore at exec (release case) === 11:02:11 (1754406131) SKIP: sanity-hsm test_30b Need 2 or more clients, have 1 SKIP 30b (2s) == sanity-hsm test 30c: Update during exec of released file must fail ========================================================== 11:02:13 (1754406133) SKIP: sanity-hsm test_30c Need 2 or more clients, have 1 SKIP 30c (1s) == sanity-hsm test 31a: Import a large file and check size during restore ========================================================== 11:02:14 (1754406134) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_31a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 39+0 records in 39+0 records out 40894464 bytes (41 MB, 39 MiB) copied, 0.500169 s, 81.8 MB/s lhsmtool_posix: 1754406135.801995 [60008]: action=1 src=d31a.sanity-hsm/f31a.sanity-hsm dst=/mnt/lustre/d31a.sanity-hsm/f31a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1754406135.804846 [60008]: importing '/mnt/lustre/d31a.sanity-hsm/f31a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_31a//d31a.sanity-hsm/f31a.sanity-hsm' lhsmtool_posix: 1754406135.820160 [60008]: imported '/mnt/lustre/d31a.sanity-hsm/f31a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_31a//040c/0x200000407:0xb:0x0'=='/tmp/arc1/sanity-hsm.test_31a//d31a.sanity-hsm/f31a.sanity-hsm' lhsmtool_posix: 1754406135.820218 [60008]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_31a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' size seen is right: 40894464 == 40894464 done PASS 31a (14s) == sanity-hsm test 31b: Restore a large unaligned file and check size during restore ========================================================== 11:02:28 (1754406148) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_31b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 39+0 records in 39+0 records out 39000000 bytes (39 MB, 37 MiB) copied, 0.420483 s, 92.8 MB/s Waiting 200s for 'SUCCEED' size seen is right: 39000000 == 39000000 done PASS 31b (15s) == sanity-hsm test 31c: Restore a large aligned file and check size during restore ========================================================== 11:02:43 (1754406163) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_31c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 39+0 records in 39+0 records out 40894464 bytes (41 MB, 39 MiB) copied, 0.434576 s, 94.1 MB/s Waiting 200s for 'SUCCEED' size seen is right: 40894464 == 40894464 done PASS 31c (16s) == sanity-hsm test 33: Kill a restore waiting process ==== 11:02:59 (1754406179) 0+0 records in 0+0 records out 0 bytes copied, 0.00238563 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_33/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Copytool is suspended on oleg305-client.virtnet Waiting 200s for 'STARTED' Copytool is continued on oleg305-client.virtnet /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh: line 2829: 62037 Terminated md5sum $f > /dev/null PASS 33 (7s) == sanity-hsm test 34: Remove file during restore ======== 11:03:06 (1754406186) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_34/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes copied, 0.00266307 s, 0.0 kB/s Copytool is suspended on oleg305-client.virtnet Waiting 200s for 'STARTED' Copytool is continued on oleg305-client.virtnet /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh: line 2861: kill: (62705) - No such process PASS 34 (5s) == sanity-hsm test 35: Overwrite file during restore ===== 11:03:11 (1754406191) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_35/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes copied, 0.0044201 s, 0.0 kB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Copytool is suspended on oleg305-client.virtnet Copytool is continued on oleg305-client.virtnet /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh: line 2898: kill: (63397) - No such process PASS 35 (6s) == sanity-hsm test 36: Move file during restore ========== 11:03:17 (1754406197) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_36/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes copied, 0.00220808 s, 0.0 kB/s Copytool is suspended on oleg305-client.virtnet Waiting 200s for 'STARTED' Updated after 2s: want 'STARTED' got 'STARTED' Copytool is continued on oleg305-client.virtnet /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh: line 2934: kill: (64071) - No such process PASS 36 (6s) == sanity-hsm test 37: re-archive a dirty file =========== 11:03:23 (1754406203) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_37/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704532 s, 14.9 MB/s Waiting 200s for 'SUCCEED' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253758 s, 41.3 MB/s Waiting 200s for 'SUCCEED' Changed after 2s: from 'SUCCEED WAITING' to 'SUCCEED SUCCEED' Updated after 3s: want 'SUCCEED' got 'SUCCEED' PASS 37 (70s) == sanity-hsm test 40: Parallel archive requests ========= 11:04:33 (1754406273) oleg305-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/hsm/max_requests=1152921504606846976: Cannot allocate memory oleg305-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/max_requests'='1152921504606846976': Cannot allocate memory pdsh@oleg305-client: oleg305-server: ssh exited with exit code 12 oleg305-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0001/hsm/max_requests=1152921504606846976: Cannot allocate memory oleg305-server: error: set_param: setting 'mdt/lustre-MDT0001/hsm/max_requests'='1152921504606846976': Cannot allocate memory pdsh@oleg305-client: oleg305-server: ssh exited with exit code 12 set max_requests=1152921504606846976 failed with 12 [ 963.715455] Lustre: lustre-MDT0000: No more memory to set HSM max_requests=1152921504606846976 (max request memory: 461MB, current total 6/975941): rc = -12 Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_40/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge Wait for all requests being enqueued...100 archive requests submitted 100 archive requests submitted 100 archive requests submitted 100 archive requests submitted OK pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 PASS 40 (95s) == sanity-hsm test 50: Archive with large number of pending HSM actions ========================================================== 11:06:08 (1754406368) oleg305-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/hsm/max_requests=1000000: Cannot allocate memory oleg305-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/max_requests'='1000000': Cannot allocate memory pdsh@oleg305-client: oleg305-server: ssh exited with exit code 12 oleg305-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0001/hsm/max_requests=1000000: Cannot allocate memory oleg305-server: error: set_param: setting 'mdt/lustre-MDT0001/hsm/max_requests'='1000000': Cannot allocate memory pdsh@oleg305-client: oleg305-server: ssh exited with exit code 12 Filesystem Inodes IUsed IFree IUse% Mounted on 192.168.203.105@tcp:/lustre 524226 1219 523007 1% /mnt/lustre - create 3426 (time 1754406380.88 total 10.00 last 342.56) - create 6615 (time 1754406390.88 total 20.00 last 318.83) - create 9662 (time 1754406400.88 total 30.00 last 304.69) total: 10000 create in 30.97 seconds: 322.86 ops/second 10000 - unlinked 0 (time 1754406441 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second Start Phase files_num: 10000 time: 37 - create 4161 (time 1754406479.38 total 10.00 last 416.01) - create 7444 (time 1754406489.38 total 20.00 last 328.26) - create 10000 (time 1754406496.90 total 27.52 last 340.05) - create 13459 (time 1754406506.90 total 37.52 last 345.85) - create 16954 (time 1754406516.90 total 47.52 last 349.49) total: 20000 create in 56.38 seconds: 354.72 ops/second - unlinked 0 (time 1754406585 ; total 0 ; last 0) - unlinked 10000 (time 1754406608 ; total 23 ; last 23) total: 20000 unlinks in 48 seconds: 416.666656 unlinks/second Middle Phase files_num: 20000 time: 58 - create 5202 (time 1754406644.95 total 10.00 last 520.12) - create 9721 (time 1754406654.95 total 20.00 last 451.90) total: 10000 create in 20.76 seconds: 481.61 ops/second 40000 - unlinked 0 (time 1754406688 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second End Phase files_num: 10000 time: 30 40000 mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge PASS 50 (352s) == sanity-hsm test 52: Opened for write file on an evicted client should be set dirty ========================================================== 11:12:00 (1754406720) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_52/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409613 s, 25.6 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' multiop /mnt/lustre/d52.sanity-hsm/f52.sanity-hsm vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8192 pdsh@oleg305-client: oleg305-client: ssh exited with exit code 5 PASS 52 (8s) == sanity-hsm test 53: Opened for read file on an evicted client should not be set dirty ========================================================== 11:12:08 (1754406728) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_53/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046411 s, 22.6 MB/s multiop /mnt/lustre/d53.sanity-hsm/f53.sanity-hsm vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.8192 pdsh@oleg305-client: oleg305-client: ssh exited with exit code 95 PASS 53 (6s) == sanity-hsm test 54: Write during an archive cancels it ========================================================== 11:12:14 (1754406734) 39+0 records in 39+0 records out 39000000 bytes (39 MB, 37 MiB) copied, 0.409819 s, 95.2 MB/s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_54/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0001.hsm.policy=+NRA Waiting 200s for 'FAILED' Waiting 170s for 'FAILED' Updated after 36s: want 'FAILED' got 'FAILED' mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA PASS 54 (42s) == sanity-hsm test 55: Truncate during an archive cancels it ========================================================== 11:12:56 (1754406776) 39+0 records in 39+0 records out 39000000 bytes (39 MB, 37 MiB) copied, 0.417238 s, 93.5 MB/s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_55/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0001.hsm.policy=+NRA mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA PASS 55 (6s) == sanity-hsm test 56: Setattr during an archive is ok === 11:13:02 (1754406782) 39+0 records in 39+0 records out 39000000 bytes (39 MB, 37 MiB) copied, 0.373545 s, 104 MB/s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_56/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' Waiting 200s for 'STARTED' Updated after 2s: want 'STARTED' got 'STARTED' Waiting 200s for 'SUCCEED' Waiting 180s for 'SUCCEED' Waiting 170s for 'SUCCEED' Updated after 37s: want 'SUCCEED' got 'SUCCEED' PASS 56 (42s) == sanity-hsm test 57: Archive a file with dirty cache on another node ========================================================== 11:13:44 (1754406824) SKIP: sanity-hsm test_57 Need 2 or more clients, have 1 SKIP 57 (2s) == sanity-hsm test 58: Truncate a released file will trigger restore ========================================================== 11:13:46 (1754406826) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_58/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' truncate up from 1516 to 3032 /mnt/lustre/d58.sanity-hsm/f58.sanity-hsm: (0x0000000b) exists dirty archived, archive_id:2 truncate down from 1516 to 758 Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' /mnt/lustre/d58.sanity-hsm/f58.sanity-hsm: (0x0000000b) exists dirty archived, archive_id:2 truncate to 0 Waiting 200s for 'SUCCEED' /mnt/lustre/d58.sanity-hsm/f58.sanity-hsm: (0x0000000b) exists dirty archived, archive_id:2 PASS 58 (11s) == sanity-hsm test 59: Release stripeless file with non-zero size ========================================================== 11:13:57 (1754406837) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_59/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' PASS 59 (4s) == sanity-hsm test 60: Changing progress update interval from default ========================================================== 11:14:01 (1754406841) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.114288 s, 91.7 MB/s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_60/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 --update-interval 5 "/mnt/lustre2"' Waiting 100s for 'start copy' Expecting a progress update within 20 seconds... 5242880 bytes copied in 3 seconds. Wait for on going archive hsm action to complete Waiting 10s for 'copied' Updated after 5s: want 'copied' got 'copied' PASS 60 (14s) == sanity-hsm test 61: Waiting archive of a removed file should fail ========================================================== 11:14:15 (1754406855) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_61/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 61 (7s) == sanity-hsm test 70: Copytool logs JSON register/unregister events to FIFO ========================================================== 11:14:22 (1754406862) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_70/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --event-fifo /tmp/sanity-hsm.test_70.Lfdy/fifo "/mnt/lustre2"' Killing existing copytools on oleg305-client.virtnet copytools stopped in 0s Register/Unregister events look OK. PASS 70 (5s) == sanity-hsm test 71: Copytool logs JSON archive events to FIFO ========================================================== 11:14:27 (1754406867) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_71/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --update-interval 5 --event-fifo /tmp/sanity-hsm.test_71.yLnz/fifo "/mnt/lustre2"' mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374894 s, 28.0 MB/s Waiting 200s for 'SUCCEED' Archive events look OK. mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA PASS 71 (9s) == sanity-hsm test 72: Copytool logs JSON restore events to FIFO ========================================================== 11:14:36 (1754406876) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_72/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --update-interval 5 --event-fifo /tmp/sanity-hsm.test_72.RFBG/fifo "/mnt/lustre2"' 16+0 records in 16+0 records out 16000000 bytes (16 MB, 15 MiB) copied, 0.198173 s, 80.7 MB/s lhsmtool_posix: 1754406878.889996 [82201]: action=1 src=d72.sanity-hsm/f72.sanity-hsm dst=/mnt/lustre/d72.sanity-hsm/f72.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1754406878.894354 [82201]: importing '/mnt/lustre/d72.sanity-hsm/f72.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_72//d72.sanity-hsm/f72.sanity-hsm' lhsmtool_posix: 1754406878.925473 [82201]: imported '/mnt/lustre/d72.sanity-hsm/f72.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_72//0bc5/0x200000bd2:0x17:0x0'=='/tmp/arc1/sanity-hsm.test_72//d72.sanity-hsm/f72.sanity-hsm' lhsmtool_posix: 1754406878.925533 [82201]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Verifying released state: Waiting 200s for 'SUCCEED' Restore events look OK. PASS 72 (8s) == sanity-hsm test 90: Archive/restore a file list ======= 11:14:44 (1754406884) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_90/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge Waiting 200s for '' pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Waiting 200s for '' Changed after 1s: from 'lrh=[type=10680000 len=136 idx=4/9652] fid=[0x200000403:0xce:0x0] dfid=[0x200000403:0xce:0x0] compound/cookie=0x0/0x6892b7b6 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9653] fid=[0x200000403:0xcf:0x0] dfid=[0x200000403:0xcf:0x0] compound/cookie=0x0/0x6892b7b7 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9654] fid=[0x200000403:0xd0:0x0] dfid=[0x200000403:0xd0:0x0] compound/cookie=0x0/0x6892b7b8 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9655] fid=[0x200000403:0xd1:0x0] dfid=[0x200000403:0xd1:0x0] compound/cookie=0x0/0x6892b7b9 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9656] fid=[0x200000403:0xd2:0x0] dfid=[0x200000403:0xd2:0x0] compound/cookie=0x0/0x6892b7ba action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9657] fid=[0x200000403:0xd3:0x0] dfid=[0x200000403:0xd3:0x0] compound/cookie=0x0/0x6892b7bb action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9658] fid=[0x200000403:0xd4:0x0] dfid=[0x200000403:0xd4:0x0] compound/cookie=0x0/0x6892b7bc action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9659] fid=[0x200000403:0xd5:0x0] dfid=[0x200000403:0xd5:0x0] compound/cookie=0x0/0x6892b7bd action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9660] fid=[0x200000403:0xd6:0x0] dfid=[0x200000403:0xd6:0x0] compound/cookie=0x0/0x6892b7be action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9661] fid=[0x200000403:0xd7:0x0] dfid=[0x200000403:0xd7:0x0] compound/cookie=0x0/0x6892b7bf action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9662] fid=[0x200000403:0xd8:0x0] dfid=[0x200000403:0xd8:0x0] compound/cookie=0x0/0x6892b7c0 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9663] fid=[0x200000403:0xd9:0x0] dfid=[0x200000403:0xd9:0x0] compound/cookie=0x0/0x6892b7c1 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9664] fid=[0x200000403:0xda:0x0] dfid=[0x200000403:0xda:0x0] compound/cookie=0x0/0x6892b7c2 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9665] fid=[0x200000403:0xdb:0x0] dfid=[0x200000403:0xdb:0x0] compound/cookie=0x0/0x6892b7c3 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9666] fid=[0x200000403:0xdc:0x0] dfid=[0x200000403:0xdc:0x0] compound/cookie=0x0/0x6892b7c4 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9667] fid=[0x200000403:0xdd:0x0] dfid=[0x200000403:0xdd:0x0] compound/cookie=0x0/0x6892b7c5 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9668] fid=[0x200000403:0xde:0x0] dfid=[0x200000403:0xde:0x0] compound/cookie=0x0/0x6892b7c6 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9669] fid=[0x200000403:0xdf:0x0] dfid=[0x200000403:0xdf:0x0] compound/cookie=0x0/0x6892b7c7 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9670] fid=[0x200000403:0xe0:0x0] dfid=[0x200000403:0xe0:0x0] compound/cookie=0x0/0x6892b7c8 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9671] fid=[0x200000403:0xe1:0x0] dfid=[0x200000403:0xe1:0x0] compound/cookie=0x0/0x6892b7c9 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9672] fid=[0x200000403:0xe2:0x0] dfid=[0x200000403:0xe2:0x0] compound/cookie=0x0/0x6892b7ca action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9673] fid=[0x200000403:0xe3:0x0] dfid=[0x200000403:0xe3:0x0] compound/cookie=0x0/0x6892b7cb action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9674] fid=[0x200000403:0xe4:0x0] dfid=[0x200000403:0xe4:0x0] compound/cookie=0x0/0x6892b7cc action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9675] fid=[0x200000403:0xe5:0x0] dfid=[0x200000403:0xe5:0x0] compound/cookie=0x0/0x6892b7cd action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9676] fid=[0x200000403:0xe6:0x0] dfid=[0x200000403:0xe6:0x0] compound/cookie=0x0/0x6892b7ce action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9677] fid=[0x200000403:0xe7:0x0] dfid=[0x200000403:0xe7:0x0] compound/cookie=0x0/0x6892b7cf action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9678] fid=[0x200000403:0xe8:0x0] dfid=[0x200000403:0xe8:0x0] compound/cookie=0x0/0x6892b7d0 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9679] fid=[0x200000403:0xe9:0x0] dfid=[0x200000403:0xe9:0x0] compound/cookie=0x0/0x6892b7d1 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9680] fid=[0x200000403:0xea:0x0] dfid=[0x200000403:0xea:0x0] compound/cookie=0x0/0x6892b7d2 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9681] fid=[0x200000403:0xeb:0x0] dfid=[0x200000403:0xeb:0x0] compound/cookie=0x0/0x6892b7d3 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9682] fid=[0x200000403:0xec:0x0] dfid=[0x200000403:0xec:0x0] compound/cookie=0x0/0x6892b7d4 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9683] fid=[0x200000403:0xed:0x0] dfid=[0x200000403:0xed:0x0] compound/cookie=0x0/0x6892b7d5 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9684] fid=[0x200000403:0xee:0x0] dfid=[0x200000403:0xee:0x0] compound/cookie=0x0/0x6892b7d6 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9685] fid=[0x200000403:0xef:0x0] dfid=[0x200000403:0xef:0x0] compound/cookie=0x0/0x6892b7d7 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9686] fid=[0x200000403:0xf0:0x0] dfid=[0x200000403:0xf0:0x0] compound/cookie=0x0/0x6892b7d8 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9687] fid=[0x200000403:0xf1:0x0] dfid=[0x200000403:0xf1:0x0] compound/cookie=0x0/0x6892b7d9 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9688] fid=[0x200000403:0xf2:0x0] dfid=[0x200000403:0xf2:0x0] compound/cookie=0x0/0x6892b7da action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9689] fid=[0x200000403:0xf3:0x0] dfid=[0x200000403:0xf3:0x0] compound/cookie=0x0/0x6892b7db action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9690] fid=[0x200000403:0xf4:0x0] dfid=[0x200000403:0xf4:0x0] compound/cookie=0x0/0x6892b7dc action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9691] fid=[0x200000403:0xf5:0x0] dfid=[0x200000403:0xf5:0x0] compound/cookie=0x0/0x6892b7dd action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9692] fid=[0x200000403:0xf6:0x0] dfid=[0x200000403:0xf6:0x0] compound/cookie=0x0/0x6892b7de action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9693] fid=[0x200000403:0xf7:0x0] dfid=[0x200000403:0xf7:0x0] compound/cookie=0x0/0x6892b7df action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9694] fid=[0x200000403:0xf8:0x0] dfid=[0x200000403:0xf8:0x0] compound/cookie=0x0/0x6892b7e0 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9695] fid=[0x200000403:0xf9:0x0] dfid=[0x200000403:0xf9:0x0] compound/cookie=0x0/0x6892b7e1 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9696] fid=[0x200000403:0xfa:0x0] dfid=[0x200000403:0xfa:0x0] compound/cookie=0x0/0x6892b7e2 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9697] fid=[0x200000403:0xfb:0x0] dfid=[0x200000403:0xfb:0x0] compound/cookie=0x0/0x6892b7e3 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9698] fid=[0x200000403:0xfc:0x0] dfid=[0x200000403:0xfc:0x0] compound/cookie=0x0/0x6892b7e4 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9700] fid=[0x200000403:0xfd:0x0] dfid=[0x200000403:0xfd:0x0] compound/cookie=0x0/0x6892b7e5 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9701] fid=[0x200000403:0xfe:0x0] dfid=[0x200000403:0xfe:0x0] compound/cookie=0x0/0x6892b7e6 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9702] fid=[0x200000403:0xff:0x0] dfid=[0x200000403:0xff:0x0] compound/cookie=0x0/0x6892b7e7 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9703] fid=[0x200000403:0x100:0x0] dfid=[0x200000403:0x100:0x0] compound/cookie=0x0/0x6892b7e8 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[]' to 'lrh=[type=10680000 len=136 idx=4/9676] fid=[0x200000403:0xe6:0x0] dfid=[0x200000403:0xe6:0x0] compound/cookie=0x0/0x6892b7ce action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[] lrh=[type=10680000 len=136 idx=4/9677] fid=[0x200000403:0xe7:0x0] dfid=[0x200000403:0xe7:0x0] compound/cookie=0x0/0x6892b7cf action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[] lrh=[type=10680000 len=136 idx=4/9678] fid=[0x200000403:0xe8:0x0] dfid=[0x200000403:0xe8:0x0] compound/cookie=0x0/0x6892b7d0 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[] lrh=[type=10680000 len=136 idx=4/9679] fid=[0x200000403:0xe9:0x0] dfid=[0x200000403:0xe9:0x0] compound/cookie=0x0/0x6892b7d1 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9680] fid=[0x200000403:0xea:0x0] dfid=[0x200000403:0xea:0x0] compound/cookie=0x0/0x6892b7d2 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9681] fid=[0x200000403:0xeb:0x0] dfid=[0x200000403:0xeb:0x0] compound/cookie=0x0/0x6892b7d3 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9682] fid=[0x200000403:0xec:0x0] dfid=[0x200000403:0xec:0x0] compound/cookie=0x0/0x6892b7d4 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9683] fid=[0x200000403:0xed:0x0] dfid=[0x200000403:0xed:0x0] compound/cookie=0x0/0x6892b7d5 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9684] fid=[0x200000403:0xee:0x0] dfid=[0x200000403:0xee:0x0] compound/cookie=0x0/0x6892b7d6 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9685] fid=[0x200000403:0xef:0x0] dfid=[0x200000403:0xef:0x0] compound/cookie=0x0/0x6892b7d7 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9686] fid=[0x200000403:0xf0:0x0] dfid=[0x200000403:0xf0:0x0] compound/cookie=0x0/0x6892b7d8 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9687] fid=[0x200000403:0xf1:0x0] dfid=[0x200000403:0xf1:0x0] compound/cookie=0x0/0x6892b7d9 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9688] fid=[0x200000403:0xf2:0x0] dfid=[0x200000403:0xf2:0x0] compound/cookie=0x0/0x6892b7da action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9689] fid=[0x200000403:0xf3:0x0] dfid=[0x200000403:0xf3:0x0] compound/cookie=0x0/0x6892b7db action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9690] fid=[0x200000403:0xf4:0x0] dfid=[0x200000403:0xf4:0x0] compound/cookie=0x0/0x6892b7dc action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9691] fid=[0x200000403:0xf5:0x0] dfid=[0x200000403:0xf5:0x0] compound/cookie=0x0/0x6892b7dd action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9692] fid=[0x200000403:0xf6:0x0] dfid=[0x200000403:0xf6:0x0] compound/cookie=0x0/0x6892b7de action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9693] fid=[0x200000403:0xf7:0x0] dfid=[0x200000403:0xf7:0x0] compound/cookie=0x0/0x6892b7df action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9694] fid=[0x200000403:0xf8:0x0] dfid=[0x200000403:0xf8:0x0] compound/cookie=0x0/0x6892b7e0 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9695] fid=[0x200000403:0xf9:0x0] dfid=[0x200000403:0xf9:0x0] compound/cookie=0x0/0x6892b7e1 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9696] fid=[0x200000403:0xfa:0x0] dfid=[0x200000403:0xfa:0x0] compound/cookie=0x0/0x6892b7e2 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9697] fid=[0x200000403:0xfb:0x0] dfid=[0x200000403:0xfb:0x0] compound/cookie=0x0/0x6892b7e3 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9698] fid=[0x200000403:0xfc:0x0] dfid=[0x200000403:0xfc:0x0] compound/cookie=0x0/0x6892b7e4 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9700] fid=[0x200000403:0xfd:0x0] dfid=[0x200000403:0xfd:0x0] compound/cookie=0x0/0x6892b7e5 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9701] fid=[0x200000403:0xfe:0x0] dfid=[0x200000403:0xfe:0x0] compound/cookie=0x0/0x6892b7e6 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9702] fid=[0x200000403:0xff:0x0] dfid=[0x200000403:0xff:0x0] compound/cookie=0x0/0x6892b7e7 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=4/9703] fid=[0x200000403:0x100:0x0] dfid=[0x200000403:0x100:0x0] compound/cookie=0x0/0x6892b7e8 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[]' pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 PASS 90 (75s) == sanity-hsm test 100: Set coordinator /proc tunables === 11:15:59 (1754406959) Testing mdt.lustre-MDT0000.hsm.loop_period Set loop_period to 0 must failed oleg305-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/hsm/loop_period=0: Invalid argument oleg305-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/loop_period'='0': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0001/hsm/loop_period=0: Invalid argument oleg305-server: error: set_param: setting 'mdt/lustre-MDT0001/hsm/loop_period'='0': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 Testing mdt.lustre-MDT0000.hsm.grace_delay Set grace_delay to 0 must failed oleg305-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/hsm/grace_delay=0: Invalid argument oleg305-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/grace_delay'='0': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0001/hsm/grace_delay=0: Invalid argument oleg305-server: error: set_param: setting 'mdt/lustre-MDT0001/hsm/grace_delay'='0': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 Testing mdt.lustre-MDT0000.hsm.active_request_timeout Set active_request_timeout to 0 must failed oleg305-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/hsm/active_request_timeout=0: Invalid argument oleg305-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/active_request_timeout'='0': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0001/hsm/active_request_timeout=0: Invalid argument oleg305-server: error: set_param: setting 'mdt/lustre-MDT0001/hsm/active_request_timeout'='0': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 Testing mdt.lustre-MDT0000.hsm.max_requests Set max_requests to 0 must failed oleg305-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/hsm/max_requests=0: Invalid argument oleg305-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/max_requests'='0': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0001/hsm/max_requests=0: Invalid argument oleg305-server: error: set_param: setting 'mdt/lustre-MDT0001/hsm/max_requests'='0': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 Testing mdt.lustre-MDT0000.hsm.default_archive_id Set default_archive_id to 0 must failed oleg305-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/hsm/default_archive_id=0: Invalid argument oleg305-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/default_archive_id'='0': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 oleg305-server: error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0001/hsm/default_archive_id=0: Invalid argument oleg305-server: error: set_param: setting 'mdt/lustre-MDT0001/hsm/default_archive_id'='0': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 PASS 100 (20s) == sanity-hsm test 102: Verify coordinator control ======= 11:16:19 (1754406979) mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 102 (8s) == sanity-hsm test 103: Purge all requests =============== 11:16:27 (1754406987) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_103/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge Current requests pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 PASS 103 (7s) == sanity-hsm test 103a: Purge pending restore requests == 11:16:34 (1754406994) mdt.lustre-MDT0000.hsm.policy=-NBR mdt.lustre-MDT0001.hsm.policy=-NBR Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_103a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' (0x200000403:0x148:0x0|0x200000403:0x149:0x0|0x200000403:0x14a:0x0|0x200000403:0x14b:0x0|0x200000403:0x14c:0x0|0x200000403:0x14d:0x0|0x200000403:0x14e:0x0|0x200000403:0x14f:0x0).*action=ARCHIVE.*status=SUCCEED pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Killing existing copytools on oleg305-client.virtnet copytools stopped in 0s cat: '/mnt/lustre/.lustre/fid/0x200000403:0x148:0x0': No data available cat: '/mnt/lustre/.lustre/fid/0x200000403:0x149:0x0': No data available cat: '/mnt/lustre/.lustre/fid/0x200000403:0x14a:0x0': No data available cat: '/mnt/lustre/.lustre/fid/0x200000403:0x14b:0x0': No data available cat: '/mnt/lustre/.lustre/fid/0x200000403:0x14d:0x0': No data available cat: '/mnt/lustre/.lustre/fid/0x200000403:0x14c:0x0': No data available cat: '/mnt/lustre/.lustre/fid/0x200000403:0x14f:0x0': No data available cat: '/mnt/lustre/.lustre/fid/0x200000403:0x14e:0x0': No data available mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge PASS 103a (9s) == sanity-hsm test 104: Copy tool data field ============= 11:16:43 (1754407003) 0+0 records in 0+0 records out 0 bytes copied, 0.00288856 s, 0.0 kB/s mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge PASS 104 (6s) == sanity-hsm test 105: Restart of coordinator =========== 11:16:49 (1754407009) mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled mdt.lustre-MDT0000.hsm_control=purge mdt.lustre-MDT0001.hsm_control=purge PASS 105 (14s) == sanity-hsm test 106: Copytool register/unregister ===== 11:17:03 (1754407023) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_106/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' oleg305-client.virtnet: executing pgrep --pidfile=/var/run/lhsmtool_posix.pid --list-full hsmtool oleg305-client.virtnet: executing get_client_uuid /mnt/lustre2 found agent uuid=4a43a402-6c57-4091-9ce2-335332730d3a archive_id=ANY requests=[current:0 ok:0 errors:0] on mds1 found agent uuid=4a43a402-6c57-4091-9ce2-335332730d3a archive_id=ANY requests=[current:0 ok:0 errors:0] on mds2 oleg305-client.virtnet: 88913 Killing existing copytools on oleg305-client.virtnet copytools stopped in 0s pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 uuid not found in agent list on mds1 pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 uuid not found in agent list on mds2 Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_106/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' oleg305-client.virtnet: executing pgrep --pidfile=/var/run/lhsmtool_posix.pid --list-full hsmtool oleg305-client.virtnet: executing get_client_uuid /mnt/lustre2 found agent uuid=4a43a402-6c57-4091-9ce2-335332730d3a archive_id=ANY requests=[current:0 ok:0 errors:0] on mds1 found agent uuid=4a43a402-6c57-4091-9ce2-335332730d3a archive_id=ANY requests=[current:0 ok:0 errors:0] on mds2 PASS 106 (11s) == sanity-hsm test 107: Copytool re-register after MDS restart ========================================================== 11:17:14 (1754407034) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_107/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 11:17:16 (1754407036) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 11:17:31 (1754407051) targets are mounted 11:17:31 (1754407051) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 107 (27s) == sanity-hsm test 109: Policy display/change ============ 11:17:41 (1754407061) mdt.lustre-MDT0000.hsm.policy=+NBR mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0000.hsm.policy=-NBR mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0000.hsm.policy=NRA NBR Next set_param must failed oleg305-server: error: set_param: setting /sys/kernel/debug/lustre/mdt/lustre-MDT0000/hsm/policy=wrong: Invalid argument oleg305-server: error: set_param: setting 'mdt/lustre-MDT0000/hsm/policy'='wrong': Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 22 Back to default policy mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0001.hsm.policy=+NRA mdt.lustre-MDT0000.hsm.policy=-NBR mdt.lustre-MDT0001.hsm.policy=-NBR PASS 109 (9s) == sanity-hsm test 110a: Non blocking restore policy (import case) ========================================================== 11:17:50 (1754407070) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_110a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1754407071.323996 [92749]: action=1 src=d110a.sanity-hsm/f110a.sanity-hsm dst=/mnt/lustre/d110a.sanity-hsm/f110a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1754407071.327887 [92749]: importing '/mnt/lustre/d110a.sanity-hsm/f110a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_110a//d110a.sanity-hsm/f110a.sanity-hsm' lhsmtool_posix: 1754407071.349024 [92749]: imported '/mnt/lustre/d110a.sanity-hsm/f110a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_110a//0bb6/0x200000bd2:0x64:0x0'=='/tmp/arc1/sanity-hsm.test_110a//d110a.sanity-hsm/f110a.sanity-hsm' lhsmtool_posix: 1754407071.349096 [92749]: process finished, errs: 0 major, 0 minor, rc=0 (Success) mdt.lustre-MDT0000.hsm.policy=+NBR mdt.lustre-MDT0001.hsm.policy=+NBR md5sum: /mnt/lustre/d110a.sanity-hsm/f110a.sanity-hsm: No data available Waiting 200s for 'SUCCEED' mdt.lustre-MDT0000.hsm.policy=-NBR mdt.lustre-MDT0001.hsm.policy=-NBR PASS 110a (6s) == sanity-hsm test 110b: Non blocking restore policy (release case) ========================================================== 11:17:56 (1754407076) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_110b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' mdt.lustre-MDT0000.hsm.policy=+NBR mdt.lustre-MDT0001.hsm.policy=+NBR md5sum: /mnt/lustre/d110b.sanity-hsm/f110b.sanity-hsm: No data available mdt.lustre-MDT0000.hsm.policy=-NBR mdt.lustre-MDT0001.hsm.policy=-NBR PASS 110b (7s) == sanity-hsm test 111a: No retry policy (import case), restore will error ========================================================== 11:18:03 (1754407083) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_111a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1754407084.395948 [94153]: action=1 src=d111a.sanity-hsm/f111a.sanity-hsm dst=/mnt/lustre/d111a.sanity-hsm/f111a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1754407084.400499 [94153]: importing '/mnt/lustre/d111a.sanity-hsm/f111a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_111a//d111a.sanity-hsm/f111a.sanity-hsm' lhsmtool_posix: 1754407084.421839 [94153]: imported '/mnt/lustre/d111a.sanity-hsm/f111a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_111a//0bbb/0x200000bd2:0x69:0x0'=='/tmp/arc1/sanity-hsm.test_111a//d111a.sanity-hsm/f111a.sanity-hsm' lhsmtool_posix: 1754407084.421901 [94153]: process finished, errs: 0 major, 0 minor, rc=0 (Success) mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0001.hsm.policy=+NRA Remove from backend: 0x200000bd2:0x69:0x0 = /tmp/arc1/sanity-hsm.test_111a/0bbb/0x200000bd2:0x69:0x0 Waiting 200s for 'FAILED' mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA PASS 111a (6s) == sanity-hsm test 111b: No retry policy (release case), restore will error ========================================================== 11:18:09 (1754407089) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_111b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0001.hsm.policy=+NRA Remove from backend: 0x200000403:0x158:0x0 = /tmp/arc1/sanity-hsm.test_111b/055b/0x200000403:0x158:0x0 Waiting 200s for 'FAILED' mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0001.hsm.policy=-NRA PASS 111b (7s) == sanity-hsm test 112: State of recorded request ======== 11:18:16 (1754407096) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_112/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled /mnt/lustre/d112.sanity-hsm/f112.sanity-hsm: ARCHIVE waiting (from 0 to EOF) mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 112 (6s) == sanity-hsm test 113: wrong stat after restore ========= 11:18:22 (1754407102) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260693 s, 40.2 MB/s Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre3 Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_113/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre3"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Verifying released state: multiop /mnt/lustre/d113.sanity-hsm/f113.sanity-hsm voO_WRONLY:O_APPEND:_w4c TMPPIPE=/tmp/multiop_open_wait_pipe.8192 File: /mnt/lustre2/d113.sanity-hsm/f113.sanity-hsm Size: 1048576 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205306057051 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-08-05 11:18:22.000000000 -0400 Modify: 2025-08-05 11:18:22.000000000 -0400 Change: 2025-08-05 11:18:22.000000000 -0400 Birth: 2025-08-05 11:18:22.000000000 -0400 Stopping client oleg305-client.virtnet /mnt/lustre3 (opts:) PASS 113 (6s) == sanity-hsm test 114: Incompatible request does not set other requests as STARTED ========================================================== 11:18:28 (1754407108) 0+0 records in 0+0 records out 0 bytes copied, 0.00201702 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00207013 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_114/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 114 (6s) == sanity-hsm test 200: Register/Cancel archive ========== 11:18:34 (1754407114) 0+0 records in 0+0 records out 0 bytes copied, 0.00150569 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_200/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Copytool is suspended on oleg305-client.virtnet Waiting 200s for 'CANCELED' Copytool is continued on oleg305-client.virtnet PASS 200 (5s) == sanity-hsm test 201: Register/Cancel restore ========== 11:18:40 (1754407120) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_201/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 39+0 records in 39+0 records out 40894464 bytes (41 MB, 39 MiB) copied, 0.452279 s, 90.4 MB/s lhsmtool_posix: 1754407121.333882 [98556]: action=1 src=d201.sanity-hsm/f201.sanity-hsm dst=/mnt/lustre/d201.sanity-hsm/f201.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1754407121.337335 [98556]: importing '/mnt/lustre/d201.sanity-hsm/f201.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_201//d201.sanity-hsm/f201.sanity-hsm' lhsmtool_posix: 1754407121.357116 [98556]: imported '/mnt/lustre/d201.sanity-hsm/f201.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_201//0ba0/0x200000bd2:0x72:0x0'=='/tmp/arc1/sanity-hsm.test_201//d201.sanity-hsm/f201.sanity-hsm' lhsmtool_posix: 1754407121.357161 [98556]: process finished, errs: 0 major, 0 minor, rc=0 (Success) mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 201 (7s) == sanity-hsm test 202: Register/Cancel remove =========== 11:18:47 (1754407127) 0+0 records in 0+0 records out 0 bytes copied, 0.0025804 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_202/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Copytool is suspended on oleg305-client.virtnet Waiting 200s for 'CANCELED' PASS 202 (6s) == sanity-hsm test 220A: Changelog for archive =========== 11:18:53 (1754407133) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_220A/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' lustre-MDT0000.3 16HSM 15:18:56.585346253 2025.08.05 0x0 t=[0x200000403:0x163:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 PASS 220A (7s) == sanity-hsm test 220a: Changelog for failed archive ==== 11:19:00 (1754407140) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_220a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl2 cl2' Copytool is suspended on oleg305-client.virtnet Copytool is continued on oleg305-client.virtnet lustre-MDT0000.6 16HSM 15:19:04.066925959 2025.08.05 0x2 t=[0x200000403:0x164:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp p=[0x200000bd2:0x75:0x0] lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0001: Deregistered changelog user #2 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 PASS 220a (8s) == sanity-hsm test 221: Changelog for archive canceled === 11:19:08 (1754407148) 0+0 records in 0+0 records out 0 bytes copied, 0.00242453 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_221/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl3 cl3' Copytool is suspended on oleg305-client.virtnet Waiting 200s for 'STARTED' Copytool is continued on oleg305-client.virtnet lustre-MDT0000.8 16HSM 15:19:13.466143176 2025.08.05 0x7d t=[0x200000403:0x165:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp lustre-MDT0001: clear the changelog for cl3 of all records lustre-MDT0001: Deregistered changelog user #3 lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0000: Deregistered changelog user #3 PASS 221 (10s) == sanity-hsm test 222a: Changelog for explicit restore == 11:19:18 (1754407158) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_222a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1754407159.113571 [102410]: action=1 src=d222a.sanity-hsm/f222a.sanity-hsm dst=/mnt/lustre/d222a.sanity-hsm/f222a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1754407159.117578 [102410]: importing '/mnt/lustre/d222a.sanity-hsm/f222a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_222a//d222a.sanity-hsm/f222a.sanity-hsm' lhsmtool_posix: 1754407159.135384 [102410]: imported '/mnt/lustre/d222a.sanity-hsm/f222a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_222a//0bab/0x200000bd2:0x79:0x0'=='/tmp/arc1/sanity-hsm.test_222a//d222a.sanity-hsm/f222a.sanity-hsm' lhsmtool_posix: 1754407159.135440 [102410]: process finished, errs: 0 major, 0 minor, rc=0 (Success) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl4 cl4' Waiting 200s for 'SUCCEED' lustre-MDT0000.11 16HSM 15:19:21.758841055 2025.08.05 0x80 t=[0x200000bd2:0x79:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp p=[0x200000bd2:0x75:0x0] lustre-MDT0001: clear the changelog for cl4 of all records lustre-MDT0001: Deregistered changelog user #4 lustre-MDT0000: clear the changelog for cl4 of all records lustre-MDT0000: Deregistered changelog user #4 PASS 222a (9s) == sanity-hsm test 222b: Changelog for implicit restore == 11:19:27 (1754407167) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_222b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl5 cl5' Waiting 200s for 'SUCCEED' 4ef076ad20e5d4a2f4729d6ed41a3bf9 /mnt/lustre/d222b.sanity-hsm/f222b.sanity-hsm lustre-MDT0000.20 16HSM 15:19:32.154284522 2025.08.05 0x80 t=[0x200000403:0x167:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp lustre-MDT0001: clear the changelog for cl5 of all records lustre-MDT0001: Deregistered changelog user #5 lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0000: Deregistered changelog user #5 PASS 222b (10s) == sanity-hsm test 222c: Changelog for failed explicit restore ========================================================== 11:19:37 (1754407177) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_222c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' lhsmtool_posix: 1754407177.985370 [104052]: action=1 src=d222c.sanity-hsm/f222c.sanity-hsm dst=/mnt/lustre/d222c.sanity-hsm/f222c.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1754407177.988164 [104052]: importing '/mnt/lustre/d222c.sanity-hsm/f222c.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_222c//d222c.sanity-hsm/f222c.sanity-hsm' lhsmtool_posix: 1754407178.005680 [104052]: imported '/mnt/lustre/d222c.sanity-hsm/f222c.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_222c//0baf/0x200000bd2:0x7d:0x0'=='/tmp/arc1/sanity-hsm.test_222c//d222c.sanity-hsm/f222c.sanity-hsm' lhsmtool_posix: 1754407178.005789 [104052]: process finished, errs: 0 major, 0 minor, rc=0 (Success) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl6 cl6' Copytool is suspended on oleg305-client.virtnet Copytool is continued on oleg305-client.virtnet lustre-MDT0000.22 16HSM 15:19:40.458214966 2025.08.05 0x82 t=[0x200000bd2:0x7d:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp p=[0x200000bd2:0x7c:0x0] lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0001: Deregistered changelog user #6 lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0000: Deregistered changelog user #6 PASS 222c (8s) == sanity-hsm test 222d: Changelog for failed implicit restore ========================================================== 11:19:45 (1754407185) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_222d/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl7 cl7' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Remove from backend: 0x200000403:0x169:0x0 = /tmp/arc1/sanity-hsm.test_222d/056a/0x200000403:0x169:0x0 md5sum: /mnt/lustre/d222d.sanity-hsm/f222d.sanity-hsm: No data available lustre-MDT0000.29 16HSM 15:19:49.763683209 2025.08.05 0x82 t=[0x200000403:0x169:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp lustre-MDT0001: clear the changelog for cl7 of all records lustre-MDT0001: Deregistered changelog user #7 lustre-MDT0000: clear the changelog for cl7 of all records lustre-MDT0000: Deregistered changelog user #7 PASS 222d (9s) == sanity-hsm test 223a: Changelog for restore canceled (import case) ========================================================== 11:19:54 (1754407194) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_223a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' 39+0 records in 39+0 records out 40894464 bytes (41 MB, 39 MiB) copied, 0.453588 s, 90.2 MB/s mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl8 cl8' lhsmtool_posix: 1754407197.531715 [105798]: action=1 src=d223a.sanity-hsm/f223a.sanity-hsm dst=/mnt/lustre/d223a.sanity-hsm/f223a.sanity-hsm mount_point=/mnt/lustre2 lhsmtool_posix: 1754407197.537008 [105798]: importing '/mnt/lustre/d223a.sanity-hsm/f223a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_223a//d223a.sanity-hsm/f223a.sanity-hsm' lhsmtool_posix: 1754407197.550785 [105798]: imported '/mnt/lustre/d223a.sanity-hsm/f223a.sanity-hsm' from '/tmp/arc1/sanity-hsm.test_223a//0b50/0x200000bd2:0x82:0x0'=='/tmp/arc1/sanity-hsm.test_223a//d223a.sanity-hsm/f223a.sanity-hsm' lhsmtool_posix: 1754407197.550845 [105798]: process finished, errs: 0 major, 0 minor, rc=0 (Success) Waiting 200s for 'STARTED' Waiting 200s for 'CANCELED' Waiting 200s for 'SUCCEED' Waiting 190s for 'SUCCEED' Waiting 180s for 'SUCCEED' Updated after 29s: want 'SUCCEED' got 'SUCCEED' lustre-MDT0000.36 16HSM 15:20:28.401317410 2025.08.05 0xfd t=[0x200000bd2:0x82:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp lustre-MDT0001: clear the changelog for cl8 of all records lustre-MDT0001: Deregistered changelog user #8 lustre-MDT0000: clear the changelog for cl8 of all records lustre-MDT0000: Deregistered changelog user #8 PASS 223a (39s) == sanity-hsm test 223b: Changelog for restore canceled (release case) ========================================================== 11:20:33 (1754407233) 0+0 records in 0+0 records out 0 bytes copied, 0.00202231 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_223b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --bandwidth 1 "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl9 cl9' Copytool is suspended on oleg305-client.virtnet Waiting 200s for 'STARTED' Updated after 2s: want 'STARTED' got 'STARTED' Copytool is continued on oleg305-client.virtnet lustre-MDT0000.43 16HSM 15:20:38.924141229 2025.08.05 0xfd t=[0x200000403:0x16d:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp p=[0x200000bd2:0x75:0x0] lustre-MDT0001: clear the changelog for cl9 of all records lustre-MDT0001: Deregistered changelog user #9 lustre-MDT0000: clear the changelog for cl9 of all records lustre-MDT0000: Deregistered changelog user #9 PASS 223b (10s) == sanity-hsm test 224A: Changelog for remove ============ 11:20:43 (1754407243) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_224A/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl10 cl10' Waiting 200s for 'SUCCEED' lustre-MDT0000.48 16HSM 15:20:47.156925629 2025.08.05 0x200 t=[0x200000403:0x16f:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp lustre-MDT0001: clear the changelog for cl10 of all records lustre-MDT0001: Deregistered changelog user #10 lustre-MDT0000: clear the changelog for cl10 of all records lustre-MDT0000: Deregistered changelog user #10 PASS 224A (8s) == sanity-hsm test 224a: Changelog for failed remove ===== 11:20:52 (1754407252) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_224a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl11 cl11' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Remove from backend: 0x200000403:0x170:0x0 = /tmp/arc1/sanity-hsm.test_224a/0573/0x200000403:0x170:0x0 Copytool is suspended on oleg305-client.virtnet Waiting 200s for 'STARTED' Copytool is continued on oleg305-client.virtnet lustre-MDT0000.53 16HSM 15:20:58.091590646 2025.08.05 0x202 t=[0x200000403:0x170:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp p=[0x200000bd2:0x86:0x0] lustre-MDT0001: clear the changelog for cl11 of all records lustre-MDT0001: Deregistered changelog user #11 lustre-MDT0000: clear the changelog for cl11 of all records lustre-MDT0000: Deregistered changelog user #11 PASS 224a (10s) == sanity-hsm test 225: Changelog for remove canceled ==== 11:21:02 (1754407262) Test disabled PASS 225 (2s) == sanity-hsm test 226: changelog for last rm/mv with exiting archive ========================================================== 11:21:05 (1754407265) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_226/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 0x200000403:0x173:0x0 mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl12 cl12' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' lustre-MDT0000.54 16HSM 15:21:08.334533517 2025.08.05 0x280 t=[0x200000403:0x171:0x0] ef=0x13 u=0:0 nid=0@<0:0> lustre-MDT0000.55 16HSM 15:21:08.371127952 2025.08.05 0x280 t=[0x200000403:0x171:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp p=[0x200000bd2:0x80:0x0] lustre-MDT0000.56 16HSM 15:21:08.371237417 2025.08.05 0x0 t=[0x200000403:0x171:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp p=[0x200000bd2:0x80:0x0] lustre-MDT0000.57 16HSM 15:21:09.422453481 2025.08.05 0x280 t=[0x200000403:0x172:0x0] ef=0x13 u=0:0 nid=0@<0:0> lustre-MDT0000.58 16HSM 15:21:09.448943751 2025.08.05 0x280 t=[0x200000403:0x172:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp p=[0x200000bd2:0x75:0x0] lustre-MDT0000.59 16HSM 15:21:09.449000479 2025.08.05 0x0 t=[0x200000403:0x172:0x0] j=lhsmtool_posix.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp p=[0x200000bd2:0x75:0x0] lustre-MDT0000.60 06UNLNK 15:21:09.660553728 2025.08.05 0x3 t=[0x200000403:0x171:0x0] j=rm.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp p=[0x200000bd2:0x87:0x0] f226.sanity-hsm-1 lustre-MDT0000.60 06UNLNK 15:21:09.660553728 2025.08.05 0x3 t=[0x200000403:0x171:0x0] j=rm.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp p=[0x200000bd2:0x87:0x0] f226.sanity-hsm-1 lustre-MDT0000.61 08RENME 15:21:09.801156059 2025.08.05 0x3 t=[0x200000403:0x172:0x0] j=mv.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp p=[0x200000bd2:0x87:0x0] f226.sanity-hsm-2 s=[0x200000403:0x173:0x0] sp=[0x200000bd2:0x87:0x0] f226.sanity-hsm-3 lustre-MDT0001: clear the changelog for cl12 of all records lustre-MDT0001: Deregistered changelog user #12 lustre-MDT0000: clear the changelog for cl12 of all records lustre-MDT0000: Deregistered changelog user #12 PASS 226 (8s) == sanity-hsm test 227: changelog when explicit setting of HSM flags ========================================================== 11:21:13 (1754407273) 0+0 records in 0+0 records out 0 bytes copied, 0.00297006 s, 0.0 kB/s mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl13 cl13' lustre-MDT0001: clear the changelog for cl13 of all records lustre-MDT0001: Deregistered changelog user #13 lustre-MDT0000: clear the changelog for cl13 of all records lustre-MDT0000: Deregistered changelog user #13 PASS 227 (8s) == sanity-hsm test 228: On released file, return extend to FIEMAP. For [cp,tar] --sparse ========================================================== 11:21:21 (1754407281) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_228/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262452 s, 40.0 MB/s Waiting 200s for 'SUCCEED' /mnt/lustre/f228.sanity-hsm: 1 extent found tar: Removing leading `/' from member names mnt/lustre/f228.sanity-hsm PASS 228 (7s) == sanity-hsm test 250: Coordinator max request ========== 11:21:28 (1754407288) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_250/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 0+0 records in 0+0 records out 0 bytes copied, 0.00313294 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00194132 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00240345 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00328557 s, 0.0 kB/s Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' 0+0 records in 0+0 records out 0 bytes copied, 0.00294828 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.00200391 s, 0.0 kB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Copytool is suspended on oleg305-client.virtnet Waiting 200s for 'STARTED' Waiting 200s for 'STARTED' Copytool is continued on oleg305-client.virtnet PASS 250 (17s) == sanity-hsm test 251: Coordinator request timeout ====== 11:21:45 (1754407305) 0+0 records in 0+0 records out 0 bytes copied, 0.00242712 s, 0.0 kB/s mdt.lustre-MDT0000.hsm_control=disabled mdt.lustre-MDT0001.hsm_control=disabled mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_251/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Copytool is suspended on oleg305-client.virtnet Waiting 200s for 'CANCELED' Updated after 3s: want 'CANCELED' got 'CANCELED' PASS 251 (12s) == sanity-hsm test 252: Timeout'ed running archive of a removed file should be canceled ========================================================== 11:21:57 (1754407317) 0+0 records in 0+0 records out 0 bytes copied, 0.00158507 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_252/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Copytool is suspended on oleg305-client.virtnet Waiting 200s for 'STARTED' Copytool is continued on oleg305-client.virtnet PASS 252 (7s) == sanity-hsm test 253: Check for wrong file size after release ========================================================== 11:22:04 (1754407324) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_253/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 10+0 records in 10+0 records out 10000000 bytes (10 MB, 9.5 MiB) copied, 0.086037 s, 116 MB/s fail_loc=0x807 Cannot send HSM request (use of /mnt/lustre/d253.sanity-hsm/f253.sanity-hsm): Invalid argument could not release file PASS 253 (3s) == sanity-hsm test 254a: Request counters are initialized to zero ========================================================== 11:22:07 (1754407327) PASS 254a (3s) == sanity-hsm test 254b: Request counters are correctly incremented and decremented ========================================================== 11:22:11 (1754407331) Will launch 63 requests of each type Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_254b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Checking archive requests Copytool is suspended on oleg305-client.virtnet Copytool is continued on oleg305-client.virtnet Checking restore requests Copytool is suspended on oleg305-client.virtnet Copytool is continued on oleg305-client.virtnet Checking remove requests Copytool is suspended on oleg305-client.virtnet Copytool is continued on oleg305-client.virtnet PASS 254b (19s) == sanity-hsm test 255: Copytool registration wakes the coordinator up ========================================================== 11:22:30 (1754407350) 0+0 records in 0+0 records out 0 bytes copied, 0.00153583 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_255/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Killing existing copytools on oleg305-client.virtnet copytools stopped in 0s 0+0 records in 0+0 records out 0 bytes copied, 0.00193428 s, 0.0 kB/s 0x200000403:0x203:0x0 mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_255/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' PASS 255 (13s) == sanity-hsm test 260a: Restore request have priority over other requests ========================================================== 11:22:43 (1754407363) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436583 s, 24.0 MB/s 0x200000403:0x204:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400029 s, 26.2 MB/s 0x200000403:0x205:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408628 s, 25.7 MB/s 0x200000403:0x206:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433259 s, 24.2 MB/s 0x200000403:0x207:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384034 s, 27.3 MB/s 0x200000403:0x208:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377143 s, 27.8 MB/s 0x200000403:0x209:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348388 s, 30.1 MB/s 0x200000403:0x20a:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393887 s, 26.6 MB/s 0x200000403:0x20b:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038822 s, 27.0 MB/s 0x200000403:0x20c:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389968 s, 26.9 MB/s 0x200000403:0x20d:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387612 s, 27.1 MB/s 0x200000403:0x20e:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384286 s, 27.3 MB/s 0x200000403:0x20f:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377041 s, 27.8 MB/s 0x200000403:0x210:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410153 s, 25.6 MB/s 0x200000403:0x211:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379404 s, 27.6 MB/s 0x200000403:0x212:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352329 s, 29.8 MB/s 0x200000403:0x213:0x0 Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Killing existing copytools on oleg305-client.virtnet copytools stopped in 0s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' ARCHIVE ARCHIVE RESTORE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE PASS 260a (15s) == sanity-hsm test 260b: Restore request have priority over other requests ========================================================== 11:22:58 (1754407378) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396965 s, 26.4 MB/s 0x200000403:0x216:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041172 s, 25.5 MB/s 0x200000403:0x217:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036093 s, 29.1 MB/s 0x200000403:0x218:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332913 s, 31.5 MB/s 0x200000403:0x219:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409634 s, 25.6 MB/s 0x200000403:0x21a:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419714 s, 25.0 MB/s 0x200000403:0x21b:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431116 s, 24.3 MB/s 0x200000403:0x21c:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042066 s, 24.9 MB/s 0x200000403:0x21d:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380911 s, 27.5 MB/s 0x200000403:0x21e:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421613 s, 24.9 MB/s 0x200000403:0x21f:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386194 s, 27.2 MB/s 0x200000403:0x220:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378117 s, 27.7 MB/s 0x200000403:0x221:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393383 s, 26.7 MB/s 0x200000403:0x222:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347663 s, 30.2 MB/s 0x200000403:0x223:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431723 s, 24.3 MB/s 0x200000403:0x224:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399548 s, 26.2 MB/s 0x200000403:0x225:0x0 Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2 "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Killing existing copytools on oleg305-client.virtnet copytools stopped in 0s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2 "/mnt/lustre2"' Waiting 200s for 'SUCCEED' RESTORE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE PASS 260b (17s) == sanity-hsm test 260c: Requests are not reordered on the 'hot' path of the coordinator ========================================================== 11:23:15 (1754407395) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373033 s, 28.1 MB/s 0x200000403:0x227:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03412 s, 30.7 MB/s 0x200000403:0x228:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380481 s, 27.6 MB/s 0x200000403:0x229:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381476 s, 27.5 MB/s 0x200000403:0x22a:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360015 s, 29.1 MB/s 0x200000403:0x22b:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472715 s, 22.2 MB/s 0x200000403:0x22c:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042026 s, 25.0 MB/s 0x200000403:0x22d:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402585 s, 26.0 MB/s 0x200000403:0x22e:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452211 s, 23.2 MB/s 0x200000403:0x22f:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442858 s, 23.7 MB/s 0x200000403:0x230:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412546 s, 25.4 MB/s 0x200000403:0x231:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356317 s, 29.4 MB/s 0x200000403:0x232:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399698 s, 26.2 MB/s 0x200000403:0x233:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398312 s, 26.3 MB/s 0x200000403:0x234:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395592 s, 26.5 MB/s 0x200000403:0x235:0x0 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415063 s, 25.3 MB/s 0x200000403:0x236:0x0 Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2 "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Killing existing copytools on oleg305-client.virtnet copytools stopped in 0s mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_260c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2 "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE ARCHIVE RESTORE PASS 260c (20s) == sanity-hsm test 261: Report 0 bytes size after HSM release ========================================================== 11:23:35 (1754407415) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_261/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00355436 s, 2.3 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' /mnt/lustre/d261.sanity-hsm/f261.sanity-hsm: (0x00000009) exists archived, archive_id:2 Waiting 200s for 'SUCCEED' /mnt/lustre/d261.sanity-hsm/f261.sanity-hsm: (0x0000000d) released exists archived, archive_id:2 PASS 261 (6s) == sanity-hsm test 262: The client should return 1 block for HSM released files ========================================================== 11:23:41 (1754407421) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_262/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00339209 s, 2.4 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' PASS 262 (4s) == sanity-hsm test 300: On disk coordinator state kept between MDT umount/mount ========================================================== 11:23:45 (1754407425) Stop coordinator and remove coordinator state at mount mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 11:23:50 (1754407430) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 11:24:04 (1754407444) targets are mounted 11:24:04 (1754407444) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Set coordinator start at mount, and start coordinator Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 11:24:35 (1754407475) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 11:24:50 (1754407490) targets are mounted 11:24:50 (1754407490) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 300 (73s) == sanity-hsm test 301: HSM tunnable are persistent ====== 11:24:59 (1754407499) Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 11:25:02 (1754407502) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 11:25:16 (1754407516) targets are mounted 11:25:16 (1754407516) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 301 (27s) == sanity-hsm test 302: HSM tunnable are persistent when CDT is off ========================================================== 11:25:25 (1754407525) mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 11:25:30 (1754407530) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 11:25:44 (1754407544) targets are mounted 11:25:44 (1754407544) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds2 on oleg305-server Stopping /mnt/lustre-mds2 (opts:) on oleg305-server 11:25:53 (1754407553) shut down facet: mds2 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds2 to oleg305-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0001 11:26:07 (1754407567) targets are mounted 11:26:07 (1754407567) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 302 (52s) == sanity-hsm test 400: Single request is sent to the right MDT ========================================================== 11:26:17 (1754407577) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_400/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392846 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036311 s, 28.9 MB/s archive successful on mdt0 archive successful on mdt1 PASS 400 (3s) == sanity-hsm test 401: Compound requests split and sent to their respective MDTs ========================================================== 11:26:21 (1754407581) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_401/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440161 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393386 s, 26.7 MB/s Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' archive successful on mdt0 archive successful on mdt1 PASS 401 (5s) == sanity-hsm test 402a: Copytool start fails if all MDTs are inactive ========================================================== 11:26:25 (1754407585) deactivate lustre-MDT0000-mdc-ffff8df944dcf000 on oleg305-client.virtnet deactivate lustre-MDT0001-mdc-ffff8df944dcf000 on oleg305-client.virtnet deactivate lustre-MDT0000-mdc-ffff8df950d18000 on oleg305-client.virtnet deactivate lustre-MDT0001-mdc-ffff8df950d18000 on oleg305-client.virtnet Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_402a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Failed to start copytool agt1 on 'oleg305-client.virtnet' pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 uuid not found in agent list on mds1 pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 uuid not found in agent list on mds2 activate lustre-MDT0000-mdc-ffff8df944dcf000 on oleg305-client.virtnet activate lustre-MDT0001-mdc-ffff8df944dcf000 on oleg305-client.virtnet activate lustre-MDT0000-mdc-ffff8df950d18000 on oleg305-client.virtnet activate lustre-MDT0001-mdc-ffff8df950d18000 on oleg305-client.virtnet PASS 402a (4s) == sanity-hsm test 402b: CDT must retry request upon slow start of CT ========================================================== 11:26:29 (1754407589) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_402b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' fail_loc=0x14d fail_loc=0 Waiting 200s for 'SUCCEED' Updated after 4s: want 'SUCCEED' got 'SUCCEED' PASS 402b (17s) == sanity-hsm test 403: Copytool starts with inactive MDT and register on reconnect ========================================================== 11:26:46 (1754407606) deactivate lustre-MDT0001-mdc-ffff8df944dcf000 on oleg305-client.virtnet deactivate lustre-MDT0001-mdc-ffff8df950d18000 on oleg305-client.virtnet Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_403/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' oleg305-client.virtnet: executing pgrep --pidfile=/var/run/lhsmtool_posix.pid --list-full hsmtool oleg305-client.virtnet: executing get_client_uuid /mnt/lustre2 found agent uuid=4a43a402-6c57-4091-9ce2-335332730d3a archive_id=ANY requests=[current:0 ok:0 errors:0] on mds1 pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 uuid not found in agent list on mds2 oleg305-client.virtnet: 132373 activate lustre-MDT0001-mdc-ffff8df944dcf000 on oleg305-client.virtnet activate lustre-MDT0001-mdc-ffff8df950d18000 on oleg305-client.virtnet found agent uuid=4a43a402-6c57-4091-9ce2-335332730d3a archive_id=ANY requests=[current:0 ok:0 errors:0] on mds1 found agent uuid=4a43a402-6c57-4091-9ce2-335332730d3a archive_id=ANY requests=[current:0 ok:0 errors:0] on mds2 PASS 403 (7s) == sanity-hsm test 404: Inactive MDT does not block requests for active MDTs ========================================================== 11:26:53 (1754407613) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_404/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341692 s, 30.7 MB/s deactivate lustre-MDT0001-mdc-ffff8df944dcf000 on oleg305-client.virtnet deactivate lustre-MDT0001-mdc-ffff8df950d18000 on oleg305-client.virtnet Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' archive successful on mdt0 activate lustre-MDT0001-mdc-ffff8df944dcf000 on oleg305-client.virtnet activate lustre-MDT0001-mdc-ffff8df950d18000 on oleg305-client.virtnet PASS 404 (4s) == sanity-hsm test 405: archive and release under striped directory ========================================================== 11:26:57 (1754407617) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_405/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01887 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221034 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219299 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197556 s, 53.1 MB/s Waiting 200s for 'SUCCEED' archive successful on 0x240000bd2:0x1:0x0 archive successful on 0x240000bd2:0x2:0x0 archive successful on 0x240000bd2:0x3:0x0 archive successful on 0x2000032e3:0x6:0x0 PASS 405 (8s) == sanity-hsm test 406: attempting to migrate HSM archived files is safe ========================================================== 11:27:05 (1754407625) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304065 s, 34.5 MB/s old fid 0x2000032e3:0x9:0x0 Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_406/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' lfs migrate: /mnt/lustre/d406.sanity-hsm/f406.sanity-hsm migrate failed: Operation not supported (95) lfs migrate: cb_migrate_mdt_fini: error completing migration of /mnt/lustre/d406.sanity-hsm: Directory not empty (39) Waiting 200s for 'SUCCEED' new fid 0x240000bd3:0x7:0x0 Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' debug=+trace PASS 406 (8s) == sanity-hsm test 407: Check for double RESTORE records in llog ========================================================== 11:27:13 (1754407633) 0+0 records in 0+0 records out 0 bytes copied, 0.00163935 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_407/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' fail_val=5 fail_loc=0x164 Copytool is suspended on oleg305-client.virtnet mdt.lustre-MDT0000.hsm.actions= lrh=[type=10680000 len=136 idx=6/1] fid=[0x200000403:0x23d:0x0] dfid=[0x200000403:0x23d:0x0] compound/cookie=0x0/0x68922278 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/2] fid=[0x200000403:0x241:0x0] dfid=[0x200000403:0x241:0x0] compound/cookie=0x0/0x68922279 action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/3] fid=[0x2000032e2:0x2:0x0] dfid=[0x2000032e2:0x2:0x0] compound/cookie=0x0/0x6892227a action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/4] fid=[0x2000032e3:0x3:0x0] dfid=[0x2000032e3:0x3:0x0] compound/cookie=0x0/0x6892227b action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/5] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x6:0x0] compound/cookie=0x0/0x6892227c action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/6] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x7:0x0] compound/cookie=0x0/0x6892227d action=RESTORE archive#=3 flags=0x4 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/7] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x6892227e action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/8] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0xa:0x0] compound/cookie=0x0/0x6892227f action=RESTORE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/9] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x68922280 action=REMOVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/10] fid=[0x2000032e3:0xc:0x0] dfid=[0x2000032e3:0xc:0x0] compound/cookie=0x0/0x68922281 action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 11:27:22 (1754407642) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 11:27:36 (1754407656) targets are mounted 11:27:36 (1754407656) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 mdt.lustre-MDT0000.hsm.actions= lrh=[type=10680000 len=136 idx=6/3] fid=[0x2000032e2:0x2:0x0] dfid=[0x2000032e2:0x2:0x0] compound/cookie=0x0/0x6892227a action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/4] fid=[0x2000032e3:0x3:0x0] dfid=[0x2000032e3:0x3:0x0] compound/cookie=0x0/0x6892227b action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/5] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x6:0x0] compound/cookie=0x0/0x6892227c action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/6] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x7:0x0] compound/cookie=0x0/0x6892227d action=RESTORE archive#=3 flags=0x4 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/7] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x6892227e action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/8] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0xa:0x0] compound/cookie=0x0/0x6892227f action=RESTORE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/9] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x68922280 action=REMOVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/10] fid=[0x2000032e3:0xc:0x0] dfid=[0x2000032e3:0xc:0x0] compound/cookie=0x0/0x68922281 action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/11] fid=[0x2000032e3:0xc:0x0] dfid=[0x2000032e3:0xc:0x0] compound/cookie=0x0/0x68922282 action=RESTORE archive#=2 flags=0x4 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[] Copytool is continued on oleg305-client.virtnet d41d8cd98f00b204e9800998ecf8427e /mnt/lustre/d407.sanity-hsm/f407.sanity-hsm d41d8cd98f00b204e9800998ecf8427e /mnt/lustre2/d407.sanity-hsm/f407.sanity-hsm pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 mdt.lustre-MDT0000.hsm.actions= lrh=[type=10680000 len=136 idx=6/4] fid=[0x2000032e3:0x3:0x0] dfid=[0x2000032e3:0x3:0x0] compound/cookie=0x0/0x6892227b action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/5] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x6:0x0] compound/cookie=0x0/0x6892227c action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/6] fid=[0x2000032e3:0x6:0x0] dfid=[0x2000032e3:0x7:0x0] compound/cookie=0x0/0x6892227d action=RESTORE archive#=3 flags=0x4 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/7] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x6892227e action=ARCHIVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/8] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0xa:0x0] compound/cookie=0x0/0x6892227f action=RESTORE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/9] fid=[0x2000032e3:0x9:0x0] dfid=[0x2000032e3:0x9:0x0] compound/cookie=0x0/0x68922280 action=REMOVE archive#=3 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/10] fid=[0x2000032e3:0xc:0x0] dfid=[0x2000032e3:0xc:0x0] compound/cookie=0x0/0x68922281 action=ARCHIVE archive#=2 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] lrh=[type=10680000 len=136 idx=6/11] fid=[0x2000032e3:0xc:0x0] dfid=[0x2000032e3:0xd:0x0] compound/cookie=0x0/0x68922282 action=RESTORE archive#=2 flags=0x4 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] PASS 407 (35s) == sanity-hsm test 408: Verify fiemap on release file ==== 11:27:48 (1754407668) Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_408/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0160793 s, 4.1 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0112032 s, 5.8 MB/s File: /mnt/lustre/f408.sanity-hsm Size: 262144 Blocks: 256 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115406615871500 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-08-05 11:27:49.000000000 -0400 Modify: 2025-08-05 11:27:49.000000000 -0400 Change: 2025-08-05 11:27:49.000000000 -0400 Birth: 2025-08-05 11:27:49.000000000 -0400 disk usage: 131072 /mnt/lustre/f408.sanity-hsm file size: 262144 /mnt/lustre/f408.sanity-hsm Waiting 200s for 'SUCCEED' extent 0 in offset 0, length 262144 flags (0x3): LAST UNKNOWN No unwritten extents, extents number 1, sum of lengths 262144, expected sum 262144 PASS 408 (6s) == sanity-hsm test 409a: Coordinator should not stop when in use ========================================================== 11:27:54 (1754407674) 0+0 records in 0+0 records out 0 bytes copied, 0.00182346 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_409a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' fail_val=5 fail_loc=0x164 cat: /mnt/lustre/d409a.sanity-hsm/f409a.sanity-hsm: No data available mdt.lustre-MDT0000.hsm_control=shutdown mdt.lustre-MDT0001.hsm_control=shutdown mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled Killing existing copytools on oleg305-client.virtnet copytools stopped in 0s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_409a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' mdt.lustre-MDT0000.hsm_control=enabled mdt.lustre-MDT0001.hsm_control=enabled PASS 409a (15s) == sanity-hsm test 409b: getattr released file with CDT stopped after remount ========================================================== 11:28:09 (1754407689) 0+0 records in 0+0 records out 0 bytes copied, 0.00229627 s, 0.0 kB/s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_409b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' Killing existing copytools on oleg305-client.virtnet copytools stopped in 0s Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 11:28:34 (1754407714) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 11:28:49 (1754407729) targets are mounted 11:28:49 (1754407729) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec File: /mnt/lustre/d409b.sanity-hsm/f409b.sanity-hsm Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115406632648721 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-08-05 11:28:10.000000000 -0400 Modify: 2025-08-05 11:28:10.000000000 -0400 Change: 2025-08-05 11:28:10.000000000 -0400 Birth: 2025-08-05 11:28:09.000000000 -0400 PASS 409b (69s) == sanity-hsm test 410: lfs data_version -s allows release of force-archived file ========================================================== 11:29:18 (1754407758) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661415 s, 15.9 MB/s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_410/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Cannot send HSM request (use of /mnt/lustre/d410.sanity-hsm/f410.sanity-hsm): Operation not permitted PASS 410 (2s) == sanity-hsm test 411: hsm_ops rbac role ================ 11:29:20 (1754407760) On MGS 192.168.203.105, active = nodemap.active=1 waiting 10 secs for sync On MGS 192.168.203.105, test_411.trusted_nodemap = nodemap.test_411.trusted_nodemap=1 waiting 10 secs for sync Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_411/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' /mnt/lustre/d411.sanity-hsm/fileA: (0x00000000) Waiting 200s for 'SUCCEED' Waiting 200s for 'SUCCEED' Updated after 2s: want 'SUCCEED' got 'SUCCEED' Waiting 200s for 'SUCCEED' Killing existing copytools on oleg305-client.virtnet copytools stopped in 0s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_411/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' On MGS 192.168.203.105, test_411.rbac = nodemap.test_411.rbac=file_perms,dne_ops,quota_ops,byfid_ops,chlg_ops,fscrypt_admin,server_upcall,ignore_root_prjquota,local_admin waiting 10 secs for sync /mnt/lustre/d411.sanity-hsm/fileB: (0x00000000) Cannot send HSM request (use of /mnt/lustre/d411.sanity-hsm/fileB): Permission denied Cannot send HSM request (use of /mnt/lustre/d411.sanity-hsm/fileB): Operation not permitted Cannot send HSM request (use of /mnt/lustre/d411.sanity-hsm/fileB): Permission denied Cannot send HSM request (use of /mnt/lustre/d411.sanity-hsm/fileB): Permission denied lfs hsm_set: change hsm flags for '/mnt/lustre/d411.sanity-hsm/fileB' failed: Permission denied lfs hsm_clear: change hsm flags for '/mnt/lustre/d411.sanity-hsm/fileB' failed: Permission denied On MGS 192.168.203.105, active = nodemap.active=0 waiting 10 secs for sync oleg305-server: error: nodemap 'test_411' does not exist oleg305-server: nodemap_del: Invalid argument pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 On MGS 192.168.203.105, active = nodemap.active=0 waiting 10 secs for sync PASS 411 (67s) == sanity-hsm test 500: various LLAPI HSM tests ========== 11:30:27 (1754407827) striped dir -i0 -c2 -H all_char /mnt/lustre/d500.sanity-hsm test1 @1754407843: Register/unregister copytool 2000x to check for leaks ...... pass test2 @1754407843: Re/un-register copytool multiple times without error ....... pass test3 @1754407843: Pass bad parameters to llapi_hsm_copytool_register() ....... pass test4 @1754407843: Bad parameters to llapi_hsm_copytool_unregister() .......... pass test5 @1754407843: Test llapi_hsm_copytool_recv() in non-blocking mode ........ pass test6 @1754407843: Test llapi_hsm_copytool_recv() with bogus parameters ....... pass test7 @1754407843: Test event polling (without actual traffic) ................ pass test50 @1754407843: Test llapi_hsm_state_get()/get_fd() ....................... pass test51 @1754407843: Test llapi_hsm_state_set_fd() ............................. pass test52 @1754407843: Test llapi_hsm_current_action() ........................... pass test100 @1754407844: Simple archive creation with no progress reported. ....... pass test101 @1754407846: Simple archive creation with progress every byte. ........ pass test102 @1754407848: Archive creation with progress every byte, backwards ..... pass test103 @1754407849: Archive creation with a single progress report ........... pass test104 @1754407850: Archive creation with two progress reports ............... pass test105 @1754407851: Archive creation with one bogus progress report .......... pass test106 @1754407852: Archive creation with one empty progress report .......... pass test107 @1754407853: Archive creation with one bogus progress report .......... pass test108 @1754407855: Archive creation with same progress report each time ..... pass test109 @1754407856: Archive creation with one report, with large number ...... pass test110 @1754407857: Archive with 10 progress reports, checking progress ...... pass test111 @1754407858: Archive with 10 reports in reverse, checking progress .... pass test112 @1754407859: Archive with 10 reports, duplicated, check progress ...... pass test113 @1754407860: Archive with 9 reports, with 20% overlapping coverage .... pass PASS 500 (35s) == sanity-hsm test 600: Changelog fields 'u=' and 'nid=' ========================================================== 11:31:02 (1754407862) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d600.sanity-hsm/f600.sanity-hsm] Got NID '192.168.203.5@tcp' lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 PASS 600 (6s) == sanity-hsm test 601: OPEN Changelog entry ============= 11:31:09 (1754407869) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl2 cl2' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0001.4 10OPEN 15:31:11.450744243 2025.08.05 0x1 t=[0x240000bd3:0xa:0x0] j=cat.0 ef=0x17 u=0:0 nid=192.168.203.5@tcp m=r-- p=[0x240000bd3:0x9:0x0] lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0001: Deregistered changelog user #2 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 PASS 601 (5s) == sanity-hsm test 602: Changelog record CLOSE only if open+write or OPEN recorded ========================================================== 11:31:15 (1754407875) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl3 cl3' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0001: clear the changelog for cl3 of all records lustre-MDT0000.9 11CLOSE 15:31:17.625928325 2025.08.05 0x1 t=[0x2000032e2:0x2e:0x0] j=cat.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0001: clear the changelog for cl3 of all records lustre-MDT0000.10 10OPEN 15:31:17.733544085 2025.08.05 0x242 t=[0x2000032e2:0x2e:0x0] j=bash.0 ef=0x17 u=0:0 nid=192.168.203.5@tcp m=-w- lustre-MDT0000.11 23GXATR 15:31:17.735892566 2025.08.05 0x0 t=[0x2000032e2:0x2e:0x0] j=bash.0 ef=0x1b u=0:0 nid=192.168.203.5@tcp x=user.job lustre-MDT0000.12 13TRUNC 15:31:17.737573921 2025.08.05 0xe t=[0x2000032e2:0x2e:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp p=[0x2000032e2:0x2d:0x0] lustre-MDT0000.13 11CLOSE 15:31:17.745301216 2025.08.05 0x242 t=[0x2000032e2:0x2e:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp lustre-MDT0000.13 11CLOSE 15:31:17.745301216 2025.08.05 0x242 t=[0x2000032e2:0x2e:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp mdd.lustre-MDT0000.changelog_mask=-OPEN mdd.lustre-MDT0001.changelog_mask=-OPEN lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0001: clear the changelog for cl3 of all records f lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0001: clear the changelog for cl3 of all records lustre-MDT0000.14 13TRUNC 15:31:18.417223049 2025.08.05 0xe t=[0x2000032e2:0x2e:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp p=[0x2000032e2:0x2d:0x0] lustre-MDT0000.15 11CLOSE 15:31:18.420842498 2025.08.05 0x242 t=[0x2000032e2:0x2e:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp lustre-MDT0000.15 11CLOSE 15:31:18.420842498 2025.08.05 0x242 t=[0x2000032e2:0x2e:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp lustre-MDT0001: clear the changelog for cl3 of all records lustre-MDT0001: Deregistered changelog user #3 lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0000: Deregistered changelog user #3 PASS 602 (7s) == sanity-hsm test 603: GETXATTR Changelog entry ========= 11:31:21 (1754407881) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl4 cl4' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL lustre-MDT0000: clear the changelog for cl4 of all records lustre-MDT0001: clear the changelog for cl4 of all records getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/d603.sanity-hsm/f603.sanity-hsm user.xattr1="value1" lustre-MDT0001.10 23GXATR 15:31:24.324169911 2025.08.05 0x0 t=[0x240000bd3:0xc:0x0] j=getfattr.0 ef=0x1b u=0:0 nid=192.168.203.5@tcp x=user.xattr1 lustre-MDT0001: clear the changelog for cl4 of all records lustre-MDT0001: Deregistered changelog user #4 lustre-MDT0000: clear the changelog for cl4 of all records lustre-MDT0000: Deregistered changelog user #4 PASS 603 (6s) == sanity-hsm test 604: NOPEN Changelog entry ============ 11:31:27 (1754407887) mdd.lustre-MDT0000.changelog_deniednext=20 mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl5 cl5' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0001: clear the changelog for cl5 of all records running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cat] [/mnt/lustre2/d604.sanity-hsm/f604.sanity-hsm] cat: /mnt/lustre2/d604.sanity-hsm/f604.sanity-hsm: Permission denied lustre-MDT0000.20 24NOPEN 15:31:30.713920648 2025.08.05 0x1 t=[0x2000032e2:0x30:0x0] j=cat.500 ef=0x17 u=500:500 nid=192.168.203.5@tcp m=r-- Got NID '192.168.203.5@tcp' lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0001: clear the changelog for cl5 of all records running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cat] [/mnt/lustre2/d604.sanity-hsm/f604.sanity-hsm] cat: /mnt/lustre2/d604.sanity-hsm/f604.sanity-hsm: Permission denied lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0001: clear the changelog for cl5 of all records running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cat] [/mnt/lustre2/d604.sanity-hsm/f604.sanity-hsm] cat: /mnt/lustre2/d604.sanity-hsm/f604.sanity-hsm: Permission denied lustre-MDT0000.21 24NOPEN 15:31:51.034385932 2025.08.05 0x1 t=[0x2000032e2:0x30:0x0] j=cat.500 ef=0x17 u=500:500 nid=192.168.203.5@tcp m=r-- p=[0x2000032e2:0x2d:0x0] Got NID '192.168.203.5@tcp' lustre-MDT0001: clear the changelog for cl5 of all records lustre-MDT0001: Deregistered changelog user #5 lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0000: Deregistered changelog user #5 mdd.lustre-MDT0000.changelog_deniednext=60 PASS 604 (28s) == sanity-hsm test 605: Test OPEN and CLOSE rate limit in Changelogs ========================================================== 11:31:55 (1754407915) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl6 cl6' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0000.25 10OPEN 15:31:57.776649616 2025.08.05 0x43 t=[0x2000032e2:0x32:0x0] j=bash.0 ef=0x17 u=0:0 nid=192.168.203.5@tcp m=rw- p=[0x2000032e2:0x2b:0x0] lustre-MDT0000.25 10OPEN 15:31:57.776649616 2025.08.05 0x43 t=[0x2000032e2:0x32:0x0] j=bash.0 ef=0x17 u=0:0 nid=192.168.203.5@tcp m=rw- p=[0x2000032e2:0x2b:0x0] lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0000.26 10OPEN 15:31:58.151977486 2025.08.05 0x1 t=[0x2000032e2:0x32:0x0] j=cat.0 ef=0x17 u=0:0 nid=192.168.203.5@tcp m=r-- p=[0x2000032e2:0x2b:0x0] lustre-MDT0000.27 11CLOSE 15:31:58.158292909 2025.08.05 0x1 t=[0x2000032e2:0x32:0x0] j=cat.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp lustre-MDT0000.26 10OPEN 15:31:58.151977486 2025.08.05 0x1 t=[0x2000032e2:0x32:0x0] j=cat.0 ef=0x17 u=0:0 nid=192.168.203.5@tcp m=r-- p=[0x2000032e2:0x2b:0x0] lustre-MDT0000.27 11CLOSE 15:31:58.158292909 2025.08.05 0x1 t=[0x2000032e2:0x32:0x0] j=cat.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0001: clear the changelog for cl6 of all records running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cat] [/mnt/lustre/d605.sanity-hsm/f605.sanity-hsm] lustre-MDT0000.28 10OPEN 15:31:58.539934847 2025.08.05 0x1 t=[0x2000032e2:0x32:0x0] j=cat.500 ef=0x17 u=500:500 nid=192.168.203.5@tcp m=r-- p=[0x2000032e2:0x31:0x0] lustre-MDT0000.29 11CLOSE 15:31:58.543284026 2025.08.05 0x1 t=[0x2000032e2:0x32:0x0] j=cat.500 ef=0x13 u=500:500 nid=192.168.203.5@tcp lustre-MDT0000.28 10OPEN 15:31:58.539934847 2025.08.05 0x1 t=[0x2000032e2:0x32:0x0] j=cat.500 ef=0x17 u=500:500 nid=192.168.203.5@tcp m=r-- p=[0x2000032e2:0x31:0x0] lustre-MDT0000.29 11CLOSE 15:31:58.543284026 2025.08.05 0x1 t=[0x2000032e2:0x32:0x0] j=cat.500 ef=0x13 u=500:500 nid=192.168.203.5@tcp lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0000.30 11CLOSE 15:31:58.732278162 2025.08.05 0x43 t=[0x2000032e2:0x32:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp lustre-MDT0000.30 11CLOSE 15:31:58.732278162 2025.08.05 0x43 t=[0x2000032e2:0x32:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.203.5@tcp lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0001: Deregistered changelog user #6 lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0000: Deregistered changelog user #6 PASS 605 (7s) == sanity-hsm test 606: llog_reader groks changelog fields ========================================================== 11:32:02 (1754407922) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl7 cl7' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d606.sanity-hsm/f606.sanity-hsm] Stopping /mnt/lustre-mds1 (opts:) on oleg305-server rec #2 type=1064553b len=64 offset 8256 Header size : 8192 llh_size : 64 Time : Tue Aug 5 11:17:29 2025 Number of records: 1 cat_idx: 1 last_idx: 2 Flags: 0x2012 Target uuid : ----------------------- #02 (064)fid=[0x1:0x3b:0x0] path=O/1/d27/59 Got UID/GID 500:500 Got NID 192.168.203.5@tcp Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 lustre-MDT0001: clear the changelog for cl7 of all records lustre-MDT0001: Deregistered changelog user #7 lustre-MDT0000: clear the changelog for cl7 of all records lustre-MDT0000: Deregistered changelog user #7 PASS 606 (17s) == sanity-hsm test 607a: release a file that was migrated after being archived ========================================================== 11:32:19 (1754407939) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743396 s, 14.1 MB/s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_607a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' PASS 607a (4s) == sanity-hsm test 607b: Migrate should not change the HSM attribute of dirty files ========================================================== 11:32:23 (1754407943) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779953 s, 13.4 MB/s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_607b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' llite.lustre-ffff8df944dcf000.xattr_cache=0 llite.lustre-ffff8df950d18000.xattr_cache=0 /mnt/lustre/d607b.sanity-hsm/f607b.sanity-hsm: (0x0000000b) exists dirty archived, archive_id:1 llite.lustre-ffff8df944dcf000.xattr_cache=1 llite.lustre-ffff8df950d18000.xattr_cache=1 PASS 607b (4s) == sanity-hsm test 607c: 'lfs swap_layouts' should set dirty flag on HSM file ========================================================== 11:32:27 (1754407947) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652284 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546881 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606626 s, 17.3 MB/s Starting copytool 'agt1' on 'oleg305-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_607c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid "/mnt/lustre2"' Waiting 200s for 'SUCCEED' lfs swap_layouts: error: cannot swap layout between '/mnt/lustre/d607c.sanity-hsm/f607c.sanity-hsm-1' and '/mnt/lustre/d607c.sanity-hsm/f607c.sanity-hsm-3': Operation not permitted (1) PASS 607c (5s) == sanity-hsm test complete, duration 2488 sec =========== 11:32:32 (1754407952) === sanity-hsm: start cleanup 11:32:32 (1754407952) === Stopping clients: oleg305-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg305-client.virtnet /mnt/lustre2 opts: === sanity-hsm: finish cleanup 11:32:41 (1754407961) ===