-----============= acceptance-small: sanity-pcc ============----- Sun Feb 25 00:27:49 EST 2024
=== sanity-pcc: start setup 00:27:51 (1708838871) ===
Starting client oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre2
Started clients oleg348-client.virtnet: 
192.168.203.148@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
oleg348-client.virtnet: executing check_config_client /mnt/lustre
oleg348-client.virtnet: Checking config lustre mounted on /mnt/lustre
Checking servers environments
Checking clients oleg348-client.virtnet environments
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff88012a640800.idle_timeout=debug
osc.lustre-OST0000-osc-ffff880136a8b800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff88012a640800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff880136a8b800.idle_timeout=debug
disable quota as required
oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
=== sanity-pcc: finish setup 00:27:58 (1708838878) ===
running as uid/gid/euid/egid 500/500/500/500, groups:
 [true]
running as uid/gid/euid/egid 500/500/500/500, groups:
 [touch] [/mnt/lustre/d0_runas_test/f6705]
nobody:x:99:
excepting tests: 
Killing existing copytools on oleg348-client.virtnet
pkill: pidfile not valid
Try `pkill --help' for more information.
Set HSM on and start
Set sanity-hsm HSM policy
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 1a: Test manual lfs pcc attach with manual HSM restore ========================================================== 00:28:20 (1708838900)
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.0725256 s, 723 MB/s
mke2fs 1.46.2.wc5 (26-Mar-2022)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Discarding device blocks:  1024/51200           done                            
Creating filesystem with 51200 1k blocks and 12824 inodes
Filesystem UUID: bb533dda-1a4b-47ed-b984-f05068ac5971
Superblock backups stored on blocks: 
	8193, 24577, 40961

Allocating group tables: 0/7   done                            
Writing inode tables: 0/7   done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/7   done

/tmp/f1a.sanity-pcc: Linux rev 1.0 ext4 filesystem data, UUID=bb533dda-1a4b-47ed-b984-f05068ac5971 (extents) (64bit) (large files) (huge files)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/mnt/pcc.d1a.sanity-pcc/d1a.sanity-pcc --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
using SAVE_PROJECT_SUPPORTED=0
1+0 records in
1+0 records out
7654321 bytes (7.7 MB) copied, 0.0364104 s, 210 MB/s
Restore testing...
Waiting 200s for 'SUCCEED'
Attach and detach testing
Start to detach the /mnt/lustre/d1a.sanity-pcc/f1a.sanity-pcc
Waiting 200s for 'SUCCEED'
PASS 1a (7s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 1b: Test manual lfs pcc attach with restore on remote access ========================================================== 00:28:29 (1708838909)
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.0597663 s, 877 MB/s
mke2fs 1.46.2.wc5 (26-Mar-2022)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Discarding device blocks:  1024/51200           done                            
Creating filesystem with 51200 1k blocks and 12824 inodes
Filesystem UUID: d05ee58a-7c30-438c-8fbc-2f6597555caf
Superblock backups stored on blocks: 
	8193, 24577, 40961

Allocating group tables: 0/7   done                            
Writing inode tables: 0/7   done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/7   done

/tmp/f1b.sanity-pcc: Linux rev 1.0 ext4 filesystem data, UUID=d05ee58a-7c30-438c-8fbc-2f6597555caf (extents) (64bit) (large files) (huge files)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/mnt/pcc.d1b.sanity-pcc/d1b.sanity-pcc --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
using SAVE_PROJECT_SUPPORTED=0
1+0 records in
1+0 records out
7654321 bytes (7.7 MB) copied, 0.0364511 s, 210 MB/s
Restore testing...
Attach and detach testing
Start to detach the /mnt/lustre/d1b.sanity-pcc/f1b.sanity-pcc
Waiting 200s for 'SUCCEED'
Updated after 2s: want 'SUCCEED' got 'SUCCEED'
PASS 1b (5s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 1c: Test automated attach using Project ID with manual HSM restore ========================================================== 00:28:36 (1708838916)
using SAVE_PROJECT_SUPPORTED=0
Stopping clients: oleg348-client.virtnet /mnt/lustre (opts:)
Stopping client oleg348-client.virtnet /mnt/lustre opts:
Stopping clients: oleg348-client.virtnet /mnt/lustre2 (opts:)
Stopping client oleg348-client.virtnet /mnt/lustre2 opts:
Stopping /mnt/lustre-mds1 (opts:-f) on oleg348-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg348-server
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/nbd0 on / type ext4 (ro,relatime,stripe=32,data=ordered)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2849)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
none on /mnt type ramfs (rw,relatime)
none on /var/lib/stateless/writable type tmpfs (rw,relatime)
/dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime)
none on /var/cache/man type tmpfs (rw,relatime)
none on /var/log type tmpfs (rw,relatime)
none on /var/lib/dbus type tmpfs (rw,relatime)
none on /tmp type tmpfs (rw,relatime)
none on /var/lib/dhclient type tmpfs (rw,relatime)
none on /var/tmp type tmpfs (rw,relatime)
none on /var/lib/NetworkManager type tmpfs (rw,relatime)
none on /var/lib/systemd/random-seed type tmpfs (rw,relatime)
none on /var/spool type tmpfs (rw,relatime)
none on /var/lib/nfs type tmpfs (rw,relatime)
none on /var/lib/gssproxy type tmpfs (rw,relatime)
none on /var/lib/logrotate type tmpfs (rw,relatime)
none on /etc type tmpfs (rw,relatime)
none on /var/lib/rsyslog type tmpfs (rw,relatime)
none on /var/lib/dhclient type tmpfs (rw,relatime)
192.168.200.253:/exports/state/oleg348-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
192.168.200.253:/exports/state/oleg348-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
192.168.200.253:/exports/state/oleg348-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
192.168.200.253:/exports/testreports/40120/testresults/sanity-pcc-zfs-centos7_x86_64-centos7_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=382032k,mode=700)
/dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime)
Checking servers environments
Checking clients oleg348-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg348-server'
oleg348-server: oleg348-server.virtnet: executing load_modules_local
oleg348-server: Loading modules from /home/green/git/lustre-release/lustre
oleg348-server: detected 4 online CPUs by sysfs
oleg348-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1
Started lustre-OST0001
Starting client: oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre
Starting client oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre
Started clients oleg348-client.virtnet: 
192.168.203.148@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Starting client: oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre2
Starting client oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre2
Started clients oleg348-client.virtnet: 
192.168.203.148@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800a9d6a800.idle_timeout=debug
osc.lustre-OST0000-osc-ffff88012f885800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800a9d6a800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff88012f885800.idle_timeout=debug
disable quota as required
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.0467016 s, 1.1 GB/s
mke2fs 1.46.2.wc5 (26-Mar-2022)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Discarding device blocks:  1024/51200           done                            
Creating filesystem with 51200 1k blocks and 12824 inodes
Filesystem UUID: 8046aab6-a33e-4664-9b2b-b4ba32394f11
Superblock backups stored on blocks: 
	8193, 24577, 40961

Allocating group tables: 0/7   done                            
Writing inode tables: 0/7   done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/7   done

/tmp/f1c.sanity-pcc: Linux rev 1.0 ext4 filesystem data, UUID=8046aab6-a33e-4664-9b2b-b4ba32394f11 (extents) (64bit) (large files) (huge files)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/mnt/pcc.d1c.sanity-pcc/d1c.sanity-pcc --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
1+0 records in
1+0 records out
7654321 bytes (7.7 MB) copied, 0.0274437 s, 279 MB/s
Restore testing...
Waiting 200s for 'SUCCEED'
Attach and detach testing
Start to detach the /mnt/lustre/d1c.sanity-pcc/f1c.sanity-pcc
Waiting 200s for 'SUCCEED'
PASS 1c (36s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 1d: Test Project ID with remote access ========================================================== 00:29:14 (1708838954)
using SAVE_PROJECT_SUPPORTED=0
Stopping clients: oleg348-client.virtnet /mnt/lustre (opts:)
Stopping client oleg348-client.virtnet /mnt/lustre opts:
Stopping clients: oleg348-client.virtnet /mnt/lustre2 (opts:)
Stopping client oleg348-client.virtnet /mnt/lustre2 opts:
Stopping /mnt/lustre-mds1 (opts:-f) on oleg348-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg348-server
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/nbd0 on / type ext4 (ro,relatime,stripe=32,data=ordered)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2849)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
none on /mnt type ramfs (rw,relatime)
none on /var/lib/stateless/writable type tmpfs (rw,relatime)
/dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime)
none on /var/cache/man type tmpfs (rw,relatime)
none on /var/log type tmpfs (rw,relatime)
none on /var/lib/dbus type tmpfs (rw,relatime)
none on /tmp type tmpfs (rw,relatime)
none on /var/lib/dhclient type tmpfs (rw,relatime)
none on /var/tmp type tmpfs (rw,relatime)
none on /var/lib/NetworkManager type tmpfs (rw,relatime)
none on /var/lib/systemd/random-seed type tmpfs (rw,relatime)
none on /var/spool type tmpfs (rw,relatime)
none on /var/lib/nfs type tmpfs (rw,relatime)
none on /var/lib/gssproxy type tmpfs (rw,relatime)
none on /var/lib/logrotate type tmpfs (rw,relatime)
none on /etc type tmpfs (rw,relatime)
none on /var/lib/rsyslog type tmpfs (rw,relatime)
none on /var/lib/dhclient type tmpfs (rw,relatime)
192.168.200.253:/exports/state/oleg348-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
192.168.200.253:/exports/state/oleg348-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
192.168.200.253:/exports/state/oleg348-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
192.168.200.253:/exports/testreports/40120/testresults/sanity-pcc-zfs-centos7_x86_64-centos7_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=382032k,mode=700)
/dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime)
Checking servers environments
Checking clients oleg348-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg348-server'
oleg348-server: oleg348-server.virtnet: executing load_modules_local
oleg348-server: Loading modules from /home/green/git/lustre-release/lustre
oleg348-server: detected 4 online CPUs by sysfs
oleg348-server: Force libcfs to create 2 CPU partitions
oleg348-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Setup mgs, mdt, osts
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1
Started lustre-OST0001
Starting client: oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre
Starting client oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre
Started clients oleg348-client.virtnet: 
192.168.203.148@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Starting client: oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre2
Starting client oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre2
Started clients oleg348-client.virtnet: 
192.168.203.148@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800ab799000.idle_timeout=debug
osc.lustre-OST0000-osc-ffff8800ab79c000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800ab799000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800ab79c000.idle_timeout=debug
disable quota as required
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.042515 s, 1.2 GB/s
mke2fs 1.46.2.wc5 (26-Mar-2022)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Discarding device blocks:  1024/51200           done                            
Creating filesystem with 51200 1k blocks and 12824 inodes
Filesystem UUID: d8f8f72b-1ee5-4a04-8e21-1b12b39057f3
Superblock backups stored on blocks: 
	8193, 24577, 40961

Allocating group tables: 0/7   done                            
Writing inode tables: 0/7   done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/7   done

/tmp/f1d.sanity-pcc: Linux rev 1.0 ext4 filesystem data, UUID=d8f8f72b-1ee5-4a04-8e21-1b12b39057f3 (extents) (64bit) (large files) (huge files)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/mnt/pcc.d1d.sanity-pcc/d1d.sanity-pcc --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
1+0 records in
1+0 records out
7654321 bytes (7.7 MB) copied, 0.0349444 s, 219 MB/s
Restore testing...
Waiting 200s for 'SUCCEED'
Attach and detach testing
Start to detach the /mnt/lustre/d1d.sanity-pcc/f1d.sanity-pcc
Waiting 200s for 'SUCCEED'
PASS 1d (37s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 1e: Test RW-PCC with non-root user ==== 00:29:52 (1708838992)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/tmp/arc1/sanity-pcc.test_1e/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
/tmp/arc1/sanity-pcc.test_1e/:
  rwid: 2
  flags: 30
  autocache: projid={100}
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d1e.sanity-pcc/f1e.sanity-pcc] [bs=1024] [count=1]
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.00267082 s, 383 kB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/utils/lfs] [pcc] [attach] [-i] [2] [/mnt/lustre/d1e.sanity-pcc/f1e.sanity-pcc]
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/mnt/lustre/d1e.sanity-pcc/f1e.sanity-pcc] [of=/dev/null] [bs=1024] [count=1]
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.00030078 s, 3.4 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/tests/truncate] [/mnt/lustre/d1e.sanity-pcc/f1e.sanity-pcc] [256]
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/tests/truncate] [/mnt/lustre/d1e.sanity-pcc/f1e.sanity-pcc] [2048]
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d1e.sanity-pcc/f1e.sanity-pcc] [bs=1024] [count=1]
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.00135722 s, 754 kB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/utils/lfs] [pcc] [detach] [-k] [/mnt/lustre/d1e.sanity-pcc/f1e.sanity-pcc]
running as uid/gid/euid/egid 500/500/500/500, groups:
 [touch] [/tmp/arc1/sanity-pcc.test_1e//0007/0000/13a1/0000/0002/0000/0x2000013a1:0x7:0x0]
touch: cannot touch '/tmp/arc1/sanity-pcc.test_1e//0007/0000/13a1/0000/0002/0000/0x2000013a1:0x7:0x0': Permission denied
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/tmp/arc1/sanity-pcc.test_1e//0007/0000/13a1/0000/0002/0000/0x2000013a1:0x7:0x0] [of=/dev/null] [bs=1024] [count=1]
dd: failed to open '/tmp/arc1/sanity-pcc.test_1e//0007/0000/13a1/0000/0002/0000/0x2000013a1:0x7:0x0': Permission denied
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/tmp/arc1/sanity-pcc.test_1e//0007/0000/13a1/0000/0002/0000/0x2000013a1:0x7:0x0] [bs=1024] [count=1]
dd: failed to open '/tmp/arc1/sanity-pcc.test_1e//0007/0000/13a1/0000/0002/0000/0x2000013a1:0x7:0x0': Permission denied
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/utils/lfs] [pcc] [attach] [-i] [2] [/mnt/lustre/d1e.sanity-pcc/f1e.sanity-pcc]
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/utils/lfs] [pcc] [detach] [/mnt/lustre/d1e.sanity-pcc/f1e.sanity-pcc]
Waiting 200s for 'SUCCEED'
PASS 1e (5s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 1f: Test auto RW-PCC cache with non-root user ========================================================== 00:29:58 (1708838998)
using SAVE_PROJECT_SUPPORTED=0
Stopping clients: oleg348-client.virtnet /mnt/lustre (opts:)
Stopping client oleg348-client.virtnet /mnt/lustre opts:
Stopping clients: oleg348-client.virtnet /mnt/lustre2 (opts:)
Stopping client oleg348-client.virtnet /mnt/lustre2 opts:
Stopping /mnt/lustre-mds1 (opts:-f) on oleg348-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg348-server
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/nbd0 on / type ext4 (ro,relatime,stripe=32,data=ordered)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2849)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
none on /mnt type ramfs (rw,relatime)
none on /var/lib/stateless/writable type tmpfs (rw,relatime)
/dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime)
none on /var/cache/man type tmpfs (rw,relatime)
none on /var/log type tmpfs (rw,relatime)
none on /var/lib/dbus type tmpfs (rw,relatime)
none on /tmp type tmpfs (rw,relatime)
none on /var/lib/dhclient type tmpfs (rw,relatime)
none on /var/tmp type tmpfs (rw,relatime)
none on /var/lib/NetworkManager type tmpfs (rw,relatime)
none on /var/lib/systemd/random-seed type tmpfs (rw,relatime)
none on /var/spool type tmpfs (rw,relatime)
none on /var/lib/nfs type tmpfs (rw,relatime)
none on /var/lib/gssproxy type tmpfs (rw,relatime)
none on /var/lib/logrotate type tmpfs (rw,relatime)
none on /etc type tmpfs (rw,relatime)
none on /var/lib/rsyslog type tmpfs (rw,relatime)
none on /var/lib/dhclient type tmpfs (rw,relatime)
192.168.200.253:/exports/state/oleg348-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
192.168.200.253:/exports/state/oleg348-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
192.168.200.253:/exports/state/oleg348-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
192.168.200.253:/exports/testreports/40120/testresults/sanity-pcc-zfs-centos7_x86_64-centos7_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=382032k,mode=700)
/dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime)
Checking servers environments
Checking clients oleg348-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg348-server'
oleg348-server: oleg348-server.virtnet: executing load_modules_local
oleg348-server: Loading modules from /home/green/git/lustre-release/lustre
oleg348-server: detected 4 online CPUs by sysfs
oleg348-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1
Started lustre-OST0001
Starting client: oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre
Starting client oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre
Started clients oleg348-client.virtnet: 
192.168.203.148@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Starting client: oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre2
Starting client oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre2
Started clients oleg348-client.virtnet: 
192.168.203.148@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800ab798800.idle_timeout=debug
osc.lustre-OST0000-osc-ffff880130e6f000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800ab798800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff880130e6f000.idle_timeout=debug
disable quota as required
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.0507517 s, 1.0 GB/s
mke2fs 1.46.2.wc5 (26-Mar-2022)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Discarding device blocks:  1024/51200           done                            
Creating filesystem with 51200 1k blocks and 12824 inodes
Filesystem UUID: 8ea6b138-75b9-442f-b41f-90cb756e540f
Superblock backups stored on blocks: 
	8193, 24577, 40961

Allocating group tables: 0/7   done                            
Writing inode tables: 0/7   done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/7   done

/tmp/f1f.sanity-pcc: Linux rev 1.0 ext4 filesystem data, UUID=8ea6b138-75b9-442f-b41f-90cb756e540f (extents) (64bit) (large files) (huge files)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/mnt/pcc.d1f.sanity-pcc/d1f.sanity-pcc --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d1f.sanity-pcc/f1f.sanity-pcc] [bs=1024] [count=1]
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.00151091 s, 678 kB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/mnt/lustre/d1f.sanity-pcc/f1f.sanity-pcc] [of=/dev/null] [bs=1024] [count=1]
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.000661714 s, 1.5 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/tests/truncate] [/mnt/lustre/d1f.sanity-pcc/f1f.sanity-pcc] [256]
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/tests/truncate] [/mnt/lustre/d1f.sanity-pcc/f1f.sanity-pcc] [2048]
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d1f.sanity-pcc/f1f.sanity-pcc] [bs=256] [count=1]
1+0 records in
1+0 records out
256 bytes (256 B) copied, 0.00149096 s, 172 kB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [touch] [/mnt/pcc.d1f.sanity-pcc/d1f.sanity-pcc/0002/0000/1b71/0000/0002/0000/0x200001b71:0x2:0x0]
touch: cannot touch '/mnt/pcc.d1f.sanity-pcc/d1f.sanity-pcc/0002/0000/1b71/0000/0002/0000/0x200001b71:0x2:0x0': Permission denied
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/mnt/pcc.d1f.sanity-pcc/d1f.sanity-pcc/0002/0000/1b71/0000/0002/0000/0x200001b71:0x2:0x0] [of=/dev/null] [bs=1024] [count=1]
dd: failed to open '/mnt/pcc.d1f.sanity-pcc/d1f.sanity-pcc/0002/0000/1b71/0000/0002/0000/0x200001b71:0x2:0x0': Permission denied
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/pcc.d1f.sanity-pcc/d1f.sanity-pcc/0002/0000/1b71/0000/0002/0000/0x200001b71:0x2:0x0] [bs=1024] [count=1]
dd: failed to open '/mnt/pcc.d1f.sanity-pcc/d1f.sanity-pcc/0002/0000/1b71/0000/0002/0000/0x200001b71:0x2:0x0': Permission denied
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/utils/lfs] [pcc] [detach] [/mnt/lustre/d1f.sanity-pcc/f1f.sanity-pcc]
Waiting 200s for 'SUCCEED'
PASS 1f (35s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 1g: General permission test for RW-PCC ========================================================== 00:30:35 (1708839035)
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.0523662 s, 1.0 GB/s
mke2fs 1.46.2.wc5 (26-Mar-2022)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Discarding device blocks:  1024/51200           done                            
Creating filesystem with 51200 1k blocks and 12824 inodes
Filesystem UUID: c9f22ae1-6ce5-4cc6-a915-05feb3bebecf
Superblock backups stored on blocks: 
	8193, 24577, 40961

Allocating group tables: 0/7   done                            
Writing inode tables: 0/7   done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/7   done

/tmp/f1g.sanity-pcc: Linux rev 1.0 ext4 filesystem data, UUID=c9f22ae1-6ce5-4cc6-a915-05feb3bebecf (extents) (64bit) (large files) (huge files)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/mnt/pcc.d1g.sanity-pcc/d1g.sanity-pcc --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.00230478 s, 444 kB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/f1g.sanity-pcc] [bs=1024] [count=1]
dd: failed to open '/mnt/lustre/f1g.sanity-pcc': Permission denied
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/f1g.sanity-pcc] [bs=1024] [count=1]
dd: failed to open '/mnt/lustre/f1g.sanity-pcc': Permission denied
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/f1g.sanity-pcc] [bs=1024] [count=1]
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.00108674 s, 942 kB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/utils/lfs] [pcc] [detach] [/mnt/lustre/f1g.sanity-pcc]
detach: cannot detach '/mnt/lustre/f1g.sanity-pcc' from PCC: Operation not permitted
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/utils/lfs] [pcc] [detach] [/mnt/lustre/f1g.sanity-pcc]
Waiting 200s for 'SUCCEED'
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/mnt/lustre/f1g.sanity-pcc] [of=/dev/null] [bs=1024] [count=1]
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.00187279 s, 547 kB/s
PASS 1g (4s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 2a: Test multi open when creating ===== 00:30:40 (1708839040)
using SAVE_PROJECT_SUPPORTED=0
Stopping clients: oleg348-client.virtnet /mnt/lustre (opts:)
Stopping client oleg348-client.virtnet /mnt/lustre opts:
Stopping clients: oleg348-client.virtnet /mnt/lustre2 (opts:)
Stopping client oleg348-client.virtnet /mnt/lustre2 opts:
Stopping /mnt/lustre-mds1 (opts:-f) on oleg348-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg348-server
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/nbd0 on / type ext4 (ro,relatime,stripe=32,data=ordered)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2849)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
none on /mnt type ramfs (rw,relatime)
none on /var/lib/stateless/writable type tmpfs (rw,relatime)
/dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime)
none on /var/cache/man type tmpfs (rw,relatime)
none on /var/log type tmpfs (rw,relatime)
none on /var/lib/dbus type tmpfs (rw,relatime)
none on /tmp type tmpfs (rw,relatime)
none on /var/lib/dhclient type tmpfs (rw,relatime)
none on /var/tmp type tmpfs (rw,relatime)
none on /var/lib/NetworkManager type tmpfs (rw,relatime)
none on /var/lib/systemd/random-seed type tmpfs (rw,relatime)
none on /var/spool type tmpfs (rw,relatime)
none on /var/lib/nfs type tmpfs (rw,relatime)
none on /var/lib/gssproxy type tmpfs (rw,relatime)
none on /var/lib/logrotate type tmpfs (rw,relatime)
none on /etc type tmpfs (rw,relatime)
none on /var/lib/rsyslog type tmpfs (rw,relatime)
none on /var/lib/dhclient type tmpfs (rw,relatime)
192.168.200.253:/exports/state/oleg348-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
192.168.200.253:/exports/state/oleg348-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
192.168.200.253:/exports/state/oleg348-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
192.168.200.253:/exports/testreports/40120/testresults/sanity-pcc-zfs-centos7_x86_64-centos7_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=382032k,mode=700)
/dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime)
Checking servers environments
Checking clients oleg348-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg348-server'
oleg348-server: oleg348-server.virtnet: executing load_modules_local
oleg348-server: Loading modules from /home/green/git/lustre-release/lustre
oleg348-server: detected 4 online CPUs by sysfs
oleg348-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1
Started lustre-OST0001
Starting client: oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre
Starting client oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre
Started clients oleg348-client.virtnet: 
192.168.203.148@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Starting client: oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre2
Starting client oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre2
Started clients oleg348-client.virtnet: 
192.168.203.148@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff88012f885000.idle_timeout=debug
osc.lustre-OST0000-osc-ffff880130e6f800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff88012f885000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff880130e6f800.idle_timeout=debug
disable quota as required
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.0482962 s, 1.1 GB/s
mke2fs 1.46.2.wc5 (26-Mar-2022)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Discarding device blocks:  1024/51200           done                            
Creating filesystem with 51200 1k blocks and 12824 inodes
Filesystem UUID: 3ce3b074-fe8c-45d7-b874-84087a4be95c
Superblock backups stored on blocks: 
	8193, 24577, 40961

Allocating group tables: 0/7   done                            
Writing inode tables: 0/7   done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/7   done

/tmp/f2a.sanity-pcc: Linux rev 1.0 ext4 filesystem data, UUID=3ce3b074-fe8c-45d7-b874-84087a4be95c (extents) (64bit) (large files) (huge files)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/mnt/pcc.d2a.sanity-pcc/d2a.sanity-pcc --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
multiop /mnt/lustre/d2a.sanity-pcc/multiop vO_c
TMPPIPE=/tmp/multiop_open_wait_pipe.23797
client oleg348-client.virtnet multiop_bg started multiop_pid=23807
---------- 1 root root 14 Feb 25 00:31 /mnt/pcc.d2a.sanity-pcc/d2a.sanity-pcc/0002/0000/2341/0000/0002/0000/0x200002341:0x2:0x0
Stopping multiop_pid=23807 (kill 23807 on oleg348-client.virtnet)
PASS 2a (38s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 2b: Test multi remote open when creating ========================================================== 00:31:21 (1708839081)

 SKIP: sanity-pcc test_2b Need 2 or more clients, have 1
SKIP 2b (1s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 2c: Test multi open on different mount points when creating ========================================================== 00:31:23 (1708839083)
using SAVE_PROJECT_SUPPORTED=0
Stopping clients: oleg348-client.virtnet /mnt/lustre (opts:)
Stopping client oleg348-client.virtnet /mnt/lustre opts:
Stopping clients: oleg348-client.virtnet /mnt/lustre2 (opts:)
Stopping client oleg348-client.virtnet /mnt/lustre2 opts:
Stopping /mnt/lustre-mds1 (opts:-f) on oleg348-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg348-server
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/nbd0 on / type ext4 (ro,relatime,stripe=32,data=ordered)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2849)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
none on /mnt type ramfs (rw,relatime)
none on /var/lib/stateless/writable type tmpfs (rw,relatime)
/dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime)
none on /var/cache/man type tmpfs (rw,relatime)
none on /var/log type tmpfs (rw,relatime)
none on /var/lib/dbus type tmpfs (rw,relatime)
none on /tmp type tmpfs (rw,relatime)
none on /var/lib/dhclient type tmpfs (rw,relatime)
none on /var/tmp type tmpfs (rw,relatime)
none on /var/lib/NetworkManager type tmpfs (rw,relatime)
none on /var/lib/systemd/random-seed type tmpfs (rw,relatime)
none on /var/spool type tmpfs (rw,relatime)
none on /var/lib/nfs type tmpfs (rw,relatime)
none on /var/lib/gssproxy type tmpfs (rw,relatime)
none on /var/lib/logrotate type tmpfs (rw,relatime)
none on /etc type tmpfs (rw,relatime)
none on /var/lib/rsyslog type tmpfs (rw,relatime)
none on /var/lib/dhclient type tmpfs (rw,relatime)
192.168.200.253:/exports/state/oleg348-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
192.168.200.253:/exports/state/oleg348-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
192.168.200.253:/exports/state/oleg348-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
192.168.200.253:/exports/testreports/40120/testresults/sanity-pcc-zfs-centos7_x86_64-centos7_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=382032k,mode=700)
/dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime)
Checking servers environments
Checking clients oleg348-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg348-server'
oleg348-server: oleg348-server.virtnet: executing load_modules_local
oleg348-server: Loading modules from /home/green/git/lustre-release/lustre
oleg348-server: detected 4 online CPUs by sysfs
oleg348-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1
Started lustre-OST0001
Starting client: oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre
Starting client oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre
Started clients oleg348-client.virtnet: 
192.168.203.148@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Starting client: oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre2
Starting client oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre2
Started clients oleg348-client.virtnet: 
192.168.203.148@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800ab7de800.idle_timeout=debug
osc.lustre-OST0000-osc-ffff8800b5879800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800ab7de800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800b5879800.idle_timeout=debug
disable quota as required
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.0515887 s, 1.0 GB/s
mke2fs 1.46.2.wc5 (26-Mar-2022)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Discarding device blocks:  1024/51200           done                            
Creating filesystem with 51200 1k blocks and 12824 inodes
Filesystem UUID: d416573d-0446-434e-a3c3-e79f01a4bd9d
Superblock backups stored on blocks: 
	8193, 24577, 40961

Allocating group tables: 0/7   done                            
Writing inode tables: 0/7   done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/7   done

/tmp/f2c.sanity-pcc: Linux rev 1.0 ext4 filesystem data, UUID=d416573d-0446-434e-a3c3-e79f01a4bd9d (extents) (64bit) (large files) (huge files)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/mnt/pcc.d2c.sanity-pcc/d2c.sanity-pcc --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
multiop /mnt/lustre/d2c.sanity-pcc/f2c.sanity-pcc vO_c
TMPPIPE=/tmp/multiop_open_wait_pipe.27126
client oleg348-client.virtnet multiop_bg started multiop_pid=27136
Stopping multiop_pid=27136 (kill 27136 on oleg348-client.virtnet)
multiopen_dataPASS 2c (56s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 3a: Repeat attach/detach operations === 00:32:20 (1708839140)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/tmp/arc1/sanity-pcc.test_3a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.0021786 s, 470 kB/s
Start to attach/detach the file: /mnt/lustre/d3a.sanity-pcc/f3a.sanity-pcc
Repeat to attach/detach the same file: /mnt/lustre/d3a.sanity-pcc/f3a.sanity-pcc
PASS 3a (3s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 3b: Repeat attach/detach operations on multiple clients ========================================================== 00:32:24 (1708839144)

 SKIP: sanity-pcc test_3b Need 3 or more clients, have 1
SKIP 3b (1s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 4: Auto cache test for mmap =========== 00:32:26 (1708839146)
using SAVE_PROJECT_SUPPORTED=0
Stopping clients: oleg348-client.virtnet /mnt/lustre (opts:)
Stopping client oleg348-client.virtnet /mnt/lustre opts:
Stopping clients: oleg348-client.virtnet /mnt/lustre2 (opts:)
Stopping client oleg348-client.virtnet /mnt/lustre2 opts:
Stopping /mnt/lustre-mds1 (opts:-f) on oleg348-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg348-server
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/nbd0 on / type ext4 (ro,relatime,stripe=32,data=ordered)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2849)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
none on /mnt type ramfs (rw,relatime)
none on /var/lib/stateless/writable type tmpfs (rw,relatime)
/dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime)
none on /var/cache/man type tmpfs (rw,relatime)
none on /var/log type tmpfs (rw,relatime)
none on /var/lib/dbus type tmpfs (rw,relatime)
none on /tmp type tmpfs (rw,relatime)
none on /var/lib/dhclient type tmpfs (rw,relatime)
none on /var/tmp type tmpfs (rw,relatime)
none on /var/lib/NetworkManager type tmpfs (rw,relatime)
none on /var/lib/systemd/random-seed type tmpfs (rw,relatime)
none on /var/spool type tmpfs (rw,relatime)
none on /var/lib/nfs type tmpfs (rw,relatime)
none on /var/lib/gssproxy type tmpfs (rw,relatime)
none on /var/lib/logrotate type tmpfs (rw,relatime)
none on /etc type tmpfs (rw,relatime)
none on /var/lib/rsyslog type tmpfs (rw,relatime)
none on /var/lib/dhclient type tmpfs (rw,relatime)
192.168.200.253:/exports/state/oleg348-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
192.168.200.253:/exports/state/oleg348-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
192.168.200.253:/exports/state/oleg348-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
192.168.200.253:/exports/testreports/40120/testresults/sanity-pcc-zfs-centos7_x86_64-centos7_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=382032k,mode=700)
/dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime)
Checking servers environments
Checking clients oleg348-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg348-server'
oleg348-server: oleg348-server.virtnet: executing load_modules_local
oleg348-server: Loading modules from /home/green/git/lustre-release/lustre
oleg348-server: detected 4 online CPUs by sysfs
oleg348-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1
Started lustre-OST0001
Starting client: oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre
Starting client oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre
Started clients oleg348-client.virtnet: 
192.168.203.148@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Starting client: oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre2
Starting client oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre2
Started clients oleg348-client.virtnet: 
192.168.203.148@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800a9fe6000.idle_timeout=debug
osc.lustre-OST0000-osc-ffff8800ab7df800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800a9fe6000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800ab7df800.idle_timeout=debug
disable quota as required
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.0486572 s, 1.1 GB/s
mke2fs 1.46.2.wc5 (26-Mar-2022)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Discarding device blocks:  1024/51200           done                            
Creating filesystem with 51200 1k blocks and 12824 inodes
Filesystem UUID: 893302a0-fed4-4fba-9e08-21c9c510c784
Superblock backups stored on blocks: 
	8193, 24577, 40961

Allocating group tables: 0/7   done                            
Writing inode tables: 0/7   done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/7   done

/tmp/f4.sanity-pcc: Linux rev 1.0 ext4 filesystem data, UUID=893302a0-fed4-4fba-9e08-21c9c510c784 (extents) (64bit) (large files) (huge files)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/mnt/pcc.d4.sanity-pcc/d4.sanity-pcc --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
mmap test1: basic mmap operation (PASS, 0.005808s)
mmap test2: MAP_PRIVATE not write back (PASS, 0.00782s)
mmap test3: concurrent mmap ops on two nodes (PASS, 2.0102s)
mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 2.0233s)
mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.008163s)
mmap test6: check mmap write/read content on two nodes (SKIPPED, 0s)
mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s)
mmap test8: SIGBUS for beyond file size (SKIPPED, 0s)
mmap test9: SIGBUS for negative file offset (SKIPPED, 0s)
mmap test10: mtime not change for readonly mmap access (PASS, 1.0042s)
ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a9fe6000.lru_size=clear
ldlm.namespaces.lustre-MDT0000-mdc-ffff8800ab7df800.lru_size=clear
PASS 4 (60s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 5: Mmap & cat a RW-PCC cached file ==== 00:33:28 (1708839208)
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.0469053 s, 1.1 GB/s
mke2fs 1.46.2.wc5 (26-Mar-2022)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Discarding device blocks:  1024/51200           done                            
Creating filesystem with 51200 1k blocks and 12824 inodes
Filesystem UUID: a686b4c3-128b-4bc1-9cdb-91fbbfec10f7
Superblock backups stored on blocks: 
	8193, 24577, 40961

Allocating group tables: 0/7   done                            
Writing inode tables: 0/7   done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/7   done

/tmp/f5.sanity-pcc: Linux rev 1.0 ext4 filesystem data, UUID=a686b4c3-128b-4bc1-9cdb-91fbbfec10f7 (extents) (64bit) (large files) (huge files)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/mnt/pcc.d5.sanity-pcc/d5.sanity-pcc --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
Waiting 200s for 'SUCCEED'
PASS 5 (3s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 6: Test mmap write on RW-PCC ========== 00:33:32 (1708839212)
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.0482147 s, 1.1 GB/s
mke2fs 1.46.2.wc5 (26-Mar-2022)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Discarding device blocks:  1024/51200           done                            
Creating filesystem with 51200 1k blocks and 12824 inodes
Filesystem UUID: 5102cf91-b32a-42a8-9bf5-370daca44756
Superblock backups stored on blocks: 
	8193, 24577, 40961

Allocating group tables: 0/7   done                            
Writing inode tables: 0/7   done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/7   done

/tmp/f6.sanity-pcc: Linux rev 1.0 ext4 filesystem data, UUID=5102cf91-b32a-42a8-9bf5-370daca44756 (extents) (64bit) (large files) (huge files)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/mnt/pcc.d6.sanity-pcc/d6.sanity-pcc --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
Waiting 200s for 'SUCCEED'
PASS 6 (5s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 7a: Fake file detached between fault() and page_mkwrite() for RW-PCC ========================================================== 00:33:39 (1708839219)
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.0500554 s, 1.0 GB/s
mke2fs 1.46.2.wc5 (26-Mar-2022)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Discarding device blocks:  1024/51200           done                            
Creating filesystem with 51200 1k blocks and 12824 inodes
Filesystem UUID: 98d8fff3-5536-4ebe-8e7e-05f74c785140
Superblock backups stored on blocks: 
	8193, 24577, 40961

Allocating group tables: 0/7   done                            
Writing inode tables: 0/7   done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/7   done

/tmp/f7a.sanity-pcc: Linux rev 1.0 ext4 filesystem data, UUID=98d8fff3-5536-4ebe-8e7e-05f74c785140 (extents) (64bit) (large files) (huge files)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/mnt/pcc.d7a.sanity-pcc/d7a.sanity-pcc --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
fail_loc=0x1412
PASS 7a (2s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 7b: Test the race with concurrent mkwrite and detach ========================================================== 00:33:43 (1708839223)
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.0611188 s, 858 MB/s
mke2fs 1.46.2.wc5 (26-Mar-2022)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Discarding device blocks:  1024/51200           done                            
Creating filesystem with 51200 1k blocks and 12824 inodes
Filesystem UUID: d09f2bfd-eea3-4de3-88af-42d54a0d8557
Superblock backups stored on blocks: 
	8193, 24577, 40961

Allocating group tables: 0/7   done                            
Writing inode tables: 0/7   done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/7   done

/tmp/f7b.sanity-pcc: Linux rev 1.0 ext4 filesystem data, UUID=d09f2bfd-eea3-4de3-88af-42d54a0d8557 (extents) (64bit) (large files) (huge files)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/mnt/pcc.d7b.sanity-pcc/d7b.sanity-pcc --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
fail_loc=0x1413
fail_val=20
PASS 7b (43s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 8: Test fake -ENOSPC tolerance for RW-PCC ========================================================== 00:34:27 (1708839267)
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.0553769 s, 947 MB/s
mke2fs 1.46.2.wc5 (26-Mar-2022)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Discarding device blocks:  1024/51200           done                            
Creating filesystem with 51200 1k blocks and 12824 inodes
Filesystem UUID: 1cdef241-2fdc-4e8d-8d1c-f99539394f23
Superblock backups stored on blocks: 
	8193, 24577, 40961

Allocating group tables: 0/7   done                            
Writing inode tables: 0/7   done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/7   done

/tmp/f8.sanity-pcc: Linux rev 1.0 ext4 filesystem data, UUID=1cdef241-2fdc-4e8d-8d1c-f99539394f23 (extents) (64bit) (large files) (huge files)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/mnt/pcc.d8.sanity-pcc/d8.sanity-pcc --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
fail_loc=0x1411
PASS 8 (3s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 9: Test -ENOSPC tolerance on loop PCC device for RW-PCC ========================================================== 00:34:31 (1708839271)
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.0435889 s, 1.2 GB/s
mke2fs 1.46.2.wc5 (26-Mar-2022)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Discarding device blocks:  1024/51200           done                            
Creating filesystem with 51200 1k blocks and 12824 inodes
Filesystem UUID: c2599679-5a3a-4daf-8cba-e11632977ed8
Superblock backups stored on blocks: 
	8193, 24577, 40961

Allocating group tables: 0/7   done                            
Writing inode tables: 0/7   done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/7   done

/tmp/f9.sanity-pcc: Linux rev 1.0 ext4 filesystem data, UUID=c2599679-5a3a-4daf-8cba-e11632977ed8 (extents) (64bit) (large files) (huge files)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/mnt/pcc.9a/d9.sanity-pcc --daemon --pid-file=/var/run/lhsmtool_posix.pid  "/mnt/lustre"'
/mnt/pcc.9a/d9.sanity-pcc:
  rwid: 2
  flags: 3e
  autocache: projid={100}
60+0 records in
60+0 records out
62914560 bytes (63 MB) copied, 1.22023 s, 51.6 MB/s
PASS 9 (3s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 10a: Test RW-PCC with user quota on loop PCC device ========================================================== 00:34:36 (1708839276)
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.0503045 s, 1.0 GB/s
mke2fs 1.46.2.wc5 (26-Mar-2022)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Discarding device blocks:  1024/51200           done                            
Creating filesystem with 51200 1k blocks and 12824 inodes
Filesystem UUID: 21909a0f-c7c6-43f4-9676-4453bc8c9543
Superblock backups stored on blocks: 
	8193, 24577, 40961

Allocating group tables: 0/7   done                            
Writing inode tables: 0/7   done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/7   done

/tmp/f10a.sanity-pcc: Linux rev 1.0 ext4 filesystem data, UUID=21909a0f-c7c6-43f4-9676-4453bc8c9543 (extents) (64bit) (large files) (huge files)
*** Report for user quotas on device /dev/loop0
Block grace time: 7days; Inode grace time: 7days
                        Space limits                File limits
User            used    soft    hard  grace    used  soft  hard  grace
----------------------------------------------------------------------
root      --     13K      0K      0K              2     0     0       
sanityusr --      0K      0K  20480K              0     0     0       

Statistics:
Total blocks: 7
Data blocks: 1
Entries: 2
Used average: 2.000000

Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/mnt/pcc.d10a.sanity-pcc/d10a.sanity-pcc --daemon --pid-file=/var/run/lhsmtool_posix.pid  "/mnt/lustre"'
/mnt/pcc.d10a.sanity-pcc/d10a.sanity-pcc:
  rwid: 2
  flags: 3e
  autocache: projid={100}
15+0 records in
15+0 records out
15728640 bytes (16 MB) copied, 0.0763479 s, 206 MB/s
15+0 records in
15+0 records out
15728640 bytes (16 MB) copied, 0.0725293 s, 217 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/utils/lfs] [pcc] [attach] [-i] [2] [/mnt/lustre/d10a.sanity-pcc/uquotaA]
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/utils/lfs] [pcc] [attach] [-i] [2] [/mnt/lustre/d10a.sanity-pcc/uquotaB]
lfs pcc pcc: cannot get UNLOCK lease, ext 10: Disk quota exceeded (122)
lfs pcc pcc: cannot attach with ID: 2: Disk quota exceeded (122)
attach: cannot attach '/mnt/lustre/d10a.sanity-pcc/uquotaB' to PCC with archive ID '2': Disk quota exceeded
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d10a.sanity-pcc/uquotaA] [bs=1M] [count=30]
30+0 records in
30+0 records out
31457280 bytes (31 MB) copied, 0.552416 s, 56.9 MB/s
PASS 10a (3s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 10b: Test RW-PCC with group quota on loop PCC device ========================================================== 00:34:40 (1708839280)
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.0520203 s, 1.0 GB/s
mke2fs 1.46.2.wc5 (26-Mar-2022)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Discarding device blocks:  1024/51200           done                            
Creating filesystem with 51200 1k blocks and 12824 inodes
Filesystem UUID: 5d012751-fcb8-4809-8fbf-6ca344428935
Superblock backups stored on blocks: 
	8193, 24577, 40961

Allocating group tables: 0/7   done                            
Writing inode tables: 0/7   done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/7   done

/tmp/f10b.sanity-pcc: Linux rev 1.0 ext4 filesystem data, UUID=5d012751-fcb8-4809-8fbf-6ca344428935 (extents) (64bit) (large files) (huge files)
*** Report for group quotas on device /dev/loop0
Block grace time: 7days; Inode grace time: 7days
                        Space limits                File limits
Group           used    soft    hard  grace    used  soft  hard  grace
----------------------------------------------------------------------
root      --     13K      0K      0K              2     0     0       
sanityusr --      0K      0K  20480K              0     0     0       

Statistics:
Total blocks: 7
Data blocks: 1
Entries: 2
Used average: 2.000000

Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/mnt/pcc.d10b.sanity-pcc/d10b.sanity-pcc --daemon --pid-file=/var/run/lhsmtool_posix.pid  "/mnt/lustre"'
/mnt/pcc.d10b.sanity-pcc/d10b.sanity-pcc:
  rwid: 2
  flags: 3e
  autocache: projid={100}
15+0 records in
15+0 records out
15728640 bytes (16 MB) copied, 0.0842821 s, 187 MB/s
15+0 records in
15+0 records out
15728640 bytes (16 MB) copied, 0.0703774 s, 223 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/utils/lfs] [pcc] [attach] [-i] [2] [/mnt/lustre/d10b.sanity-pcc/gquotaA]
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/utils/lfs] [pcc] [attach] [-i] [2] [/mnt/lustre/d10b.sanity-pcc/gquotaB]
lfs pcc pcc: cannot get UNLOCK lease, ext 10: Disk quota exceeded (122)
lfs pcc pcc: cannot attach with ID: 2: Disk quota exceeded (122)
attach: cannot attach '/mnt/lustre/d10b.sanity-pcc/gquotaB' to PCC with archive ID '2': Disk quota exceeded
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d10b.sanity-pcc/gquotaA] [bs=1M] [count=30]
30+0 records in
30+0 records out
31457280 bytes (31 MB) copied, 0.586394 s, 53.6 MB/s
PASS 10b (3s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 11: Test attach fault injection with simulated PCC file path ========================================================== 00:34:44 (1708839284)
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.0483678 s, 1.1 GB/s
mke2fs 1.46.2.wc5 (26-Mar-2022)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Discarding device blocks:  1024/51200           done                            
Creating filesystem with 51200 1k blocks and 12824 inodes
Filesystem UUID: de75e002-21c8-415e-82f3-c39a840baaa9
Superblock backups stored on blocks: 
	8193, 24577, 40961

Allocating group tables: 0/7   done                            
Writing inode tables: 0/7   done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/7   done

/tmp/f11.sanity-pcc: Linux rev 1.0 ext4 filesystem data, UUID=de75e002-21c8-415e-82f3-c39a840baaa9 (extents) (64bit) (large files) (huge files)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/mnt/pcc.d11.sanity-pcc/d11.sanity-pcc --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
Lustre file: /mnt/lustre/f11.sanity-pcc LPCC dir: /mnt/pcc.d11.sanity-pcc/d11.sanity-pcc/0028/0000/32e1/0000/0002/0000
Lustre file: /mnt/lustre/f11.sanity-pcc LPCC dir: /mnt/pcc.d11.sanity-pcc/d11.sanity-pcc/002a/0000/32e1/0000/0002/0000
lfs pcc pcc: cannot get UNLOCK lease, ext 10: Permission denied (13)
lfs pcc pcc: cannot attach with ID: 2: Permission denied (13)
attach: cannot attach '/mnt/lustre/f11.sanity-pcc' to PCC with archive ID '2': Permission denied
lfs pcc pcc: cannot get UNLOCK lease, ext 10: Invalid argument (22)
lfs pcc pcc: cannot attach with ID: 2: Invalid argument (22)
attach: cannot attach '/mnt/lustre/f11.sanity-pcc' to PCC with archive ID '2': Invalid argument
PASS 11 (2s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 12: RW-PCC attach races with concurrent HSM remove ========================================================== 00:34:48 (1708839288)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/tmp/arc1/sanity-pcc.test_12/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
fail_loc=0x1414
fail_val=20
/mnt/lustre/f12.sanity-pcc: (0x0000000d) released exists archived, archive_id:2
PASS 12 (22s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 13a: Test auto RW-PCC create caching for UID/GID rule ========================================================== 00:35:11 (1708839311)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/tmp/arc1/sanity-pcc.test_13a/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
/tmp/arc1/sanity-pcc.test_13a/:
  rwid: 2
  flags: 30
  autocache: uid={500}
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d13a.sanity-pcc/f13a.sanity-pcc] [bs=1024] [count=1]
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.00119051 s, 860 kB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/mnt/lustre/d13a.sanity-pcc/f13a.sanity-pcc] [of=/dev/null] [bs=1024] [count=1]
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.000247766 s, 4.1 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/tests/truncate] [/mnt/lustre/d13a.sanity-pcc/f13a.sanity-pcc] [256]
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/tests/truncate] [/mnt/lustre/d13a.sanity-pcc/f13a.sanity-pcc] [2048]
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d13a.sanity-pcc/f13a.sanity-pcc] [bs=1024] [count=1]
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.00110562 s, 926 kB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/utils/lfs] [pcc] [detach] [/mnt/lustre/d13a.sanity-pcc/f13a.sanity-pcc]
Waiting 200s for 'SUCCEED'
/tmp/arc1/sanity-pcc.test_13a/:
  rwid: 2
  flags: 30
  autocache: gid={500}
lfs mkdir: dirstripe error on '/mnt/lustre/d13a.sanity-pcc': stripe already set
lfs setdirstripe: cannot create dir '/mnt/lustre/d13a.sanity-pcc': File exists
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d13a.sanity-pcc/f13a.sanity-pcc] [bs=1024] [count=1]
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.0011271 s, 909 kB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/mnt/lustre/d13a.sanity-pcc/f13a.sanity-pcc] [of=/dev/null] [bs=1024] [count=1]
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.000209426 s, 4.9 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/tests/truncate] [/mnt/lustre/d13a.sanity-pcc/f13a.sanity-pcc] [256]
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/tests/truncate] [/mnt/lustre/d13a.sanity-pcc/f13a.sanity-pcc] [2048]
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d13a.sanity-pcc/f13a.sanity-pcc] [bs=1024] [count=1]
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.000977921 s, 1.0 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/utils/lfs] [pcc] [detach] [/mnt/lustre/d13a.sanity-pcc/f13a.sanity-pcc]
Waiting 200s for 'SUCCEED'
PASS 13a (6s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 13b: Test auto RW-PCC create caching for file name with wildcard ========================================================== 00:35:19 (1708839319)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/tmp/arc1/sanity-pcc.test_13b/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
/tmp/arc1/sanity-pcc.test_13b/:
  rwid: 2
  flags: 30
  autocache: fname={*.h5 suffix.* Mid*dle}
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.00120531 s, 850 kB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d13b.sanity-pcc/suffix.doc] [bs=1024] [count=1]
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.0010672 s, 960 kB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d13b.sanity-pcc/MidPADdle] [bs=1024] [count=1]
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.00134136 s, 763 kB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d13b.sanity-pcc/Midpad] [bs=1024] [count=1]
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.00231642 s, 442 kB/s
PASS 13b (2s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 13c: Check auto RW-PCC create caching for UID/GID/ProjID/fname rule ========================================================== 00:35:22 (1708839322)
using SAVE_PROJECT_SUPPORTED=0
Stopping clients: oleg348-client.virtnet /mnt/lustre (opts:)
Stopping client oleg348-client.virtnet /mnt/lustre opts:
Stopping clients: oleg348-client.virtnet /mnt/lustre2 (opts:)
Stopping client oleg348-client.virtnet /mnt/lustre2 opts:
Stopping /mnt/lustre-mds1 (opts:-f) on oleg348-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg348-server
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/nbd0 on / type ext4 (ro,relatime,stripe=32,data=ordered)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2849)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
none on /mnt type ramfs (rw,relatime)
none on /var/lib/stateless/writable type tmpfs (rw,relatime)
/dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime)
none on /var/cache/man type tmpfs (rw,relatime)
none on /var/log type tmpfs (rw,relatime)
none on /var/lib/dbus type tmpfs (rw,relatime)
none on /tmp type tmpfs (rw,relatime)
none on /var/lib/dhclient type tmpfs (rw,relatime)
none on /var/tmp type tmpfs (rw,relatime)
none on /var/lib/NetworkManager type tmpfs (rw,relatime)
none on /var/lib/systemd/random-seed type tmpfs (rw,relatime)
none on /var/spool type tmpfs (rw,relatime)
none on /var/lib/nfs type tmpfs (rw,relatime)
none on /var/lib/gssproxy type tmpfs (rw,relatime)
none on /var/lib/logrotate type tmpfs (rw,relatime)
none on /etc type tmpfs (rw,relatime)
none on /var/lib/rsyslog type tmpfs (rw,relatime)
none on /var/lib/dhclient type tmpfs (rw,relatime)
192.168.200.253:/exports/state/oleg348-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
192.168.200.253:/exports/state/oleg348-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
192.168.200.253:/exports/state/oleg348-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
192.168.200.253:/exports/testreports/40120/testresults/sanity-pcc-zfs-centos7_x86_64-centos7_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.48,local_lock=none,addr=192.168.200.253)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=382032k,mode=700)
/dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime)
Checking servers environments
Checking clients oleg348-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg348-server'
oleg348-server: oleg348-server.virtnet: executing load_modules_local
oleg348-server: Loading modules from /home/green/git/lustre-release/lustre
oleg348-server: detected 4 online CPUs by sysfs
oleg348-server: Force libcfs to create 2 CPU partitions
oleg348-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
Setup mgs, mdt, osts
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1
Started lustre-OST0000
Starting ost2: -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg348-server: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1
Started lustre-OST0001
Starting client: oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre
Starting client oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre
Started clients oleg348-client.virtnet: 
192.168.203.148@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Starting client: oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre2
Starting client oleg348-client.virtnet:  -o user_xattr,flock oleg348-server@tcp:/lustre /mnt/lustre2
Started clients oleg348-client.virtnet: 
192.168.203.148@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800b6ccf800.idle_timeout=debug
osc.lustre-OST0000-osc-ffff88012a642000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800b6ccf800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff88012a642000.idle_timeout=debug
disable quota as required
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.0543451 s, 965 MB/s
mke2fs 1.46.2.wc5 (26-Mar-2022)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Discarding device blocks:  1024/51200           done                            
Creating filesystem with 51200 1k blocks and 12824 inodes
Filesystem UUID: 00ce11bf-0e37-4222-91d0-f7d169b26fa6
Superblock backups stored on blocks: 
	8193, 24577, 40961

Allocating group tables: 0/7   done                            
Writing inode tables: 0/7   done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/7   done

/tmp/f13c.sanity-pcc: Linux rev 1.0 ext4 filesystem data, UUID=00ce11bf-0e37-4222-91d0-f7d169b26fa6 (extents) (64bit) (large files) (huge files)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/mnt/pcc.d13c.sanity-pcc/d13c.sanity-pcc --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
/mnt/pcc.d13c.sanity-pcc/d13c.sanity-pcc:
  rwid: 2
  flags: 3e
  autocache: projid={100 200}&fname={*.h5},uid={500}&gid={500}
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.0019837 s, 516 kB/s
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.00120395 s, 851 kB/s
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.0020534 s, 499 kB/s
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.00122398 s, 837 kB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d13c.sanity-pcc/ugidcache] [bs=1024] [count=1]
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.001327 s, 772 kB/s
PASS 13c (34s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 14: Revocation of the layout lock should detach the file automatically ========================================================== 00:35:57 (1708839357)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/tmp/arc1/sanity-pcc.test_14/ --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b6ccf800.lru_size=clear
ldlm.namespaces.lustre-MDT0000-mdc-ffff88012a642000.lru_size=clear
PASS 14 (3s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 15: Test auto attach at open when file is still valid cached ========================================================== 00:36:01 (1708839361)
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.0438802 s, 1.2 GB/s
mke2fs 1.46.2.wc5 (26-Mar-2022)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Discarding device blocks:  1024/51200           done                            
Creating filesystem with 51200 1k blocks and 12824 inodes
Filesystem UUID: ed422abf-28c2-44b4-acdb-2015d9232f9a
Superblock backups stored on blocks: 
	8193, 24577, 40961

Allocating group tables: 0/7   done                            
Writing inode tables: 0/7   done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/7   done

/tmp/f15.sanity-pcc: Linux rev 1.0 ext4 filesystem data, UUID=ed422abf-28c2-44b4-acdb-2015d9232f9a (extents) (64bit) (large files) (huge files)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/mnt/pcc.d15.sanity-pcc/d15.sanity-pcc --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
Check open attach for non-root user
running as uid/gid/euid/egid 500/500/500/500, groups:
 [dd] [if=/dev/zero] [of=/mnt/lustre/d15.sanity-pcc/f15.sanity-pcc] [bs=1024] [count=1]
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.00208303 s, 492 kB/s
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/utils/lfs] [pcc] [attach] [-i] [2] [/mnt/lustre/d15.sanity-pcc/f15.sanity-pcc]
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/utils/lfs] [pcc] [state] [/mnt/lustre/d15.sanity-pcc/f15.sanity-pcc]
file: /mnt/lustre/d15.sanity-pcc/f15.sanity-pcc, type: readwrite, PCC file: /d15.sanity-pcc/0010/0000/3ab1/0000/0002/0000/0x200003ab1:0x10:0x0, user number: 0, flags: 0
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/utils/lfs] [pcc] [state] [/mnt/lustre/d15.sanity-pcc/f15.sanity-pcc]
ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b6ccf800.lru_size=clear
ldlm.namespaces.lustre-MDT0000-mdc-ffff88012a642000.lru_size=clear
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/utils/lfs] [pcc] [state] [/mnt/lustre/d15.sanity-pcc/f15.sanity-pcc]
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/utils/lfs] [pcc] [detach] [-k] [/mnt/lustre/d15.sanity-pcc/f15.sanity-pcc]
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/utils/lfs] [pcc] [state] [/mnt/lustre/d15.sanity-pcc/f15.sanity-pcc]
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/utils/lfs] [pcc] [detach] [/mnt/lustre/d15.sanity-pcc/f15.sanity-pcc]
check open attach for root user
ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b6ccf800.lru_size=clear
ldlm.namespaces.lustre-MDT0000-mdc-ffff88012a642000.lru_size=clear
Waiting 200s for 'SUCCEED'
PASS 15 (4s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 16: Test detach with different options ========================================================== 00:36:07 (1708839367)
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.0473124 s, 1.1 GB/s
mke2fs 1.46.2.wc5 (26-Mar-2022)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Discarding device blocks:  1024/51200           done                            
Creating filesystem with 51200 1k blocks and 12824 inodes
Filesystem UUID: 688ec1a8-3cfa-4b35-9a06-f985aa22cc74
Superblock backups stored on blocks: 
	8193, 24577, 40961

Allocating group tables: 0/7   done                            
Writing inode tables: 0/7   done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/7   done

/tmp/f16.sanity-pcc: Linux rev 1.0 ext4 filesystem data, UUID=688ec1a8-3cfa-4b35-9a06-f985aa22cc74 (extents) (64bit) (large files) (huge files)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/mnt/pcc.d16.sanity-pcc/d16.sanity-pcc --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
Test for reusing valid PCC cache
Test for the default detach
Waiting 200s for 'SUCCEED'
PASS 16 (3s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 17: Test auto attach for layout refresh ========================================================== 00:36:12 (1708839372)
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.0498646 s, 1.1 GB/s
mke2fs 1.46.2.wc5 (26-Mar-2022)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Discarding device blocks:  1024/51200           done                            
Creating filesystem with 51200 1k blocks and 12824 inodes
Filesystem UUID: 1c5d416e-8416-45d4-ae79-7185a8232b9a
Superblock backups stored on blocks: 
	8193, 24577, 40961

Allocating group tables: 0/7   done                            
Writing inode tables: 0/7   done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/7   done

/tmp/f17.sanity-pcc: Linux rev 1.0 ext4 filesystem data, UUID=1c5d416e-8416-45d4-ae79-7185a8232b9a (extents) (64bit) (large files) (huge files)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/mnt/pcc.d17.sanity-pcc/d17.sanity-pcc --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
/mnt/pcc.d17.sanity-pcc/d17.sanity-pcc:
  rwid: 2
  flags: 34
  autocache: projid={100}
Verify auto attach during IO for truncate
Verify auto attach during IO for read/write
multiop /mnt/lustre/f17.sanity-pcc vO_r
TMPPIPE=/tmp/multiop_open_wait_pipe.16387
client oleg348-client.virtnet multiop_bg started multiop_pid=16397
ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b6ccf800.lru_size=clear
ldlm.namespaces.lustre-MDT0000-mdc-ffff88012a642000.lru_size=clear
Stopping multiop_pid=16397 (kill 16397 on oleg348-client.virtnet)
PASS 17 (11s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 18: Verify size correctness after re-attach the file ========================================================== 00:36:24 (1708839384)
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.0495238 s, 1.1 GB/s
mke2fs 1.46.2.wc5 (26-Mar-2022)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Discarding device blocks:  1024/51200           done                            
Creating filesystem with 51200 1k blocks and 12824 inodes
Filesystem UUID: 2cef2f6b-2714-499e-b390-d585064c68ce
Superblock backups stored on blocks: 
	8193, 24577, 40961

Allocating group tables: 0/7   done                            
Writing inode tables: 0/7   done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/7   done

/tmp/f18.sanity-pcc: Linux rev 1.0 ext4 filesystem data, UUID=2cef2f6b-2714-499e-b390-d585064c68ce (extents) (64bit) (large files) (huge files)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/mnt/pcc.d18.sanity-pcc/d18.sanity-pcc --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
/mnt/pcc.d18.sanity-pcc/d18.sanity-pcc:
  rwid: 2
  flags: 3e
  autocache: projid={100}
4+0 records in
4+0 records out
4194304 bytes (4.2 MB) copied, 0.0453539 s, 92.5 MB/s
file: /mnt/lustre/f18.sanity-pcc, type: readwrite, PCC file: /d18.sanity-pcc/001b/0000/3ab1/0000/0002/0000/0x200003ab1:0x1b:0x0, user number: 0, flags: 0
file: /mnt/lustre/f18.sanity-pcc, type: readwrite, PCC file: /d18.sanity-pcc/001b/0000/3ab1/0000/0002/0000/0x200003ab1:0x1b:0x0, user number: 0, flags: 0
/mnt/lustre/f18.sanity-pcc has size 4194304 OK
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.00147433 s, 695 kB/s
/mnt/lustre2/f18.sanity-pcc has size 1049600 OK
file: /mnt/lustre/f18.sanity-pcc, type: none
PASS 18 (4s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 19: Verify the file re-attach works as expected ========================================================== 00:36:30 (1708839390)
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.0457879 s, 1.1 GB/s
mke2fs 1.46.2.wc5 (26-Mar-2022)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Discarding device blocks:  1024/51200           done                            
Creating filesystem with 51200 1k blocks and 12824 inodes
Filesystem UUID: a46eadfd-9143-4852-b2a5-76b03124c8f3
Superblock backups stored on blocks: 
	8193, 24577, 40961

Allocating group tables: 0/7   done                            
Writing inode tables: 0/7   done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/7   done

/tmp/f19.sanity-pcc: Linux rev 1.0 ext4 filesystem data, UUID=a46eadfd-9143-4852-b2a5-76b03124c8f3 (extents) (64bit) (large files) (huge files)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/mnt/pcc.d19.sanity-pcc/d19.sanity-pcc --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
PASS 19 (2s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 20: Auto attach works after the inode was once evicted from cache ========================================================== 00:36:34 (1708839394)
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.04702 s, 1.1 GB/s
mke2fs 1.46.2.wc5 (26-Mar-2022)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Discarding device blocks:  1024/51200           done                            
Creating filesystem with 51200 1k blocks and 12824 inodes
Filesystem UUID: 4b9d7ff5-2e3b-4f19-bb38-37e4c5252836
Superblock backups stored on blocks: 
	8193, 24577, 40961

Allocating group tables: 0/7   done                            
Writing inode tables: 0/7   done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/7   done

/tmp/f20.sanity-pcc: Linux rev 1.0 ext4 filesystem data, UUID=4b9d7ff5-2e3b-4f19-bb38-37e4c5252836 (extents) (64bit) (large files) (huge files)
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/mnt/pcc.d20.sanity-pcc/d20.sanity-pcc --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
PASS 20 (3s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 101a: Test auto attach in mount namespace (simulated container) ========================================================== 00:36:38 (1708839398)
creating user namespace for 500
sanityu+ 20026  0.0  0.0 108060   364 pts/0    S+   00:36   0:00 sleep 600
root     20028  0.0  0.0 114880   956 pts/0    S+   00:36   0:00 grep sleep
Created NS: child (sleep) pid=20026 in 0 seconds
running as uid/gid/euid/egid 500/500/500/500, groups:
 [newuidmap] [20026] [0] [500] [1]
running as uid/gid/euid/egid 500/500/500/500, groups:
 [newgidmap] [20026] [0] [500] [1]
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.0487951 s, 1.1 GB/s
mke2fs 1.46.2.wc5 (26-Mar-2022)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Discarding device blocks:  1024/51200           done                            
Creating filesystem with 51200 1k blocks and 12824 inodes
Filesystem UUID: 6ad80c6c-1080-4fd3-8ad4-12ada11aefaa
Superblock backups stored on blocks: 
	8193, 24577, 40961

Allocating group tables: 0/7   done                            
Writing inode tables: 0/7   done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/7   done

/tmp/f101a.sanity-pcc: Linux rev 1.0 ext4 filesystem data, UUID=6ad80c6c-1080-4fd3-8ad4-12ada11aefaa (extents) (64bit) (large files) (huge files)
Check for temp file in PCC mount
Check for temp file in PCC mount from inside namespace
Starting copytool 'agt1' on 'oleg348-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v1 --hsm-root=/mnt/pcc.d101a.sanity-pcc/d101a.sanity-pcc --daemon --pid-file=/var/run/lhsmtool_posix.pid --archive 2  "/mnt/lustre"'
Verify open attach from inside mount namespace
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.00245518 s, 417 kB/s
file: /mnt/lustre/d101a.sanity-pcc/f101a.sanity-pcc, type: readwrite, PCC file: /d101a.sanity-pcc/0028/0000/3ab1/0000/0002/0000/0x200003ab1:0x28:0x0, user number: 0, flags: 0
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/utils/lfs] [pcc] [state] [/mnt/lustre/d101a.sanity-pcc/f101a.sanity-pcc]
ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b6ccf800.lru_size=clear
ldlm.namespaces.lustre-MDT0000-mdc-ffff88012a642000.lru_size=clear
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/utils/lfs] [pcc] [state] [/mnt/lustre/d101a.sanity-pcc/f101a.sanity-pcc]
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/home/green/git/lustre-release/lustre/utils/lfs] [pcc] [state] [/mnt/lustre/d101a.sanity-pcc/f101a.sanity-pcc]
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.000458518 s, 2.2 MB/s
Verify auto attach at open from inside NS for RW-PCC
ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b6ccf800.lru_size=clear
ldlm.namespaces.lustre-MDT0000-mdc-ffff88012a642000.lru_size=clear
Waiting 200s for 'SUCCEED'
Updated after 2s: want 'SUCCEED' got 'SUCCEED'
PASS 101a (5s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-pcc test 102: PCC-RO should not hange for io_uring I/O engine ========================================================== 00:36:45 (1708839405)

 SKIP: sanity-pcc test_102 Client OS does not support io_uring I/O engine
SKIP 102 (1s)
debug_raw_pointers=0
debug_raw_pointers=0
== sanity-pcc test complete, duration 536 sec ============ 00:36:46 (1708839406)
=== sanity-pcc: start cleanup 00:36:47 (1708839407) ===
Stopping clients: oleg348-client.virtnet /mnt/lustre2 (opts:)
Stopping client oleg348-client.virtnet /mnt/lustre2 opts:
=== sanity-pcc: finish cleanup 00:36:47 (1708839407) ===