-----============= acceptance-small: sanity-scrub ============----- Wed May  8 18:02:20 EDT 2024
mgs: CentOS Linux release 7.9.2009 (Core)
MGS_OS_VERSION_ID=7
MGS_OS_ID=centos
MGS_OS_VERSION_CODE=117440512
MGS_OS_ID_LIKE=rhel fedora centos
mds1: CentOS Linux release 7.9.2009 (Core)
MDS1_OS_ID_LIKE=rhel fedora centos
MDS1_OS_ID=centos
MDS1_OS_VERSION_ID=7
MDS1_OS_VERSION_CODE=117440512
ost1: CentOS Linux release 7.9.2009 (Core)
OST1_OS_VERSION_CODE=117440512
OST1_OS_VERSION_ID=7
OST1_OS_ID_LIKE=rhel fedora centos
OST1_OS_ID=centos
client: CentOS Linux release 7.9.2009 (Core)
CLIENT_OS_ID=centos
CLIENT_OS_ID_LIKE=rhel fedora centos
CLIENT_OS_VERSION_ID=7
CLIENT_OS_VERSION_CODE=117440512
excepting tests: 
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg351-server'
oleg351-server: oleg351-server.virtnet: executing load_modules_local
oleg351-server: Loading modules from /home/green/git/lustre-release/lustre
oleg351-server: detected 4 online CPUs by sysfs
oleg351-server: Force libcfs to create 2 CPU partitions
oleg351-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
client=34553605 MDS=34553605 OSS=34553605
Stopping clients: oleg351-client.virtnet /mnt/lustre (opts:)
Stopping client oleg351-client.virtnet /mnt/lustre opts:
Stopping clients: oleg351-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg351-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg351-server
unloading modules on: 'oleg351-server'
oleg351-server: oleg351-server.virtnet: executing unload_modules_local
modules unloaded.
=== sanity-scrub: start setup 18:03:04 (1715205784) ===
Stopping clients: oleg351-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg351-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2
oleg351-server: oleg351-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
../libcfs/libcfs/libcfs options: 'cpu_npartitions=2'
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg351-server'
oleg351-server: oleg351-server.virtnet: executing load_modules_local
oleg351-server: Loading modules from /home/green/git/lustre-release/lustre
oleg351-server: detected 4 online CPUs by sysfs
oleg351-server: Force libcfs to create 2 CPU partitions
oleg351-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg351-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/vdc
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Format mds2: /dev/vdd
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Format ost1: /dev/vde
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Format ost2: /dev/vdf
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Checking servers environments
Checking clients oleg351-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg351-server'
oleg351-server: oleg351-server.virtnet: executing load_modules_local
oleg351-server: Loading modules from /home/green/git/lustre-release/lustre
oleg351-server: detected 4 online CPUs by sysfs
oleg351-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Commit the device label on /dev/vdc
Started lustre-MDT0000
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Commit the device label on /dev/vdd
Started lustre-MDT0001
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Commit the device label on /dev/vde
Started lustre-OST0000
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Commit the device label on /dev/vdf
Started lustre-OST0001
Starting client: oleg351-client.virtnet:  -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre
Starting client oleg351-client.virtnet:  -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre
Started clients oleg351-client.virtnet: 
192.168.203.151@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800b636f800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800b636f800.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
Updated after 4s: want 'procname_uid' got 'procname_uid'
disable quota as required
osd-ldiskfs.track_declares_assert=1
=== sanity-scrub: finish setup 18:04:04 (1715205844) ===
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-scrub test 0: Do not auto trigger OI scrub for non-backup/restore case ========================================================== 18:04:05 (1715205845)
preparing... Wed May  8 18:04:05 EDT 2024
creating 0 files on mds1
striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d0.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0105101 s, 99.8 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H all_char /mnt/lustre/d0.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00928287 s, 113 MB/s
prepared Wed May  8 18:04:07 EDT 2024.
stop mds1
stop mds2
starting MDTs without disabling OI scrub
oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Starting client: oleg351-client.virtnet:  -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre
PASS 0 (29s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-scrub test 1a: Auto trigger initial OI scrub when server mounts ========================================================== 18:04:36 (1715205876)
preparing... Wed May  8 18:04:37 EDT 2024
creating 0 files on mds1
striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d1a.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0105799 s, 99.1 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H all_char /mnt/lustre/d1a.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00971404 s, 108 MB/s
prepared Wed May  8 18:04:38 EDT 2024.
stop mds1
stop mds2
start mds1 without disabling OI scrub: -o localrecov -o user_xattr
oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Starting client: oleg351-client.virtnet:  -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre
fail_loc=0x193
fail_loc=0
192.168.203.151@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0
Stopping client oleg351-client.virtnet /mnt/lustre (opts:)
stop mds1
start mds1 with disabling OI scrub: -o localrecov -o user_xattr,noscrub
oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
PASS 1a (33s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-scrub test 1b: Trigger OI scrub when MDT mounts for OI files remove/recreate case ========================================================== 18:05:10 (1715205910)
Checking servers environments
Checking clients oleg351-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg351-server'
oleg351-server: oleg351-server.virtnet: executing load_modules_local
oleg351-server: Loading modules from /home/green/git/lustre-release/lustre
oleg351-server: detected 4 online CPUs by sysfs
oleg351-server: Force libcfs to create 2 CPU partitions
oleg351-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg351-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg351-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg351-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg351-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg351-client.virtnet:  -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre
Starting client oleg351-client.virtnet:  -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre
Started clients oleg351-client.virtnet: 
192.168.203.151@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800ac969000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800ac969000.idle_timeout=debug
disable quota as required
preparing... Wed May  8 18:05:30 EDT 2024
creating 0 files on mds1
striped dir -i0 -c1 -H crush /mnt/lustre/d1b.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00958189 s, 109 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H crush2 /mnt/lustre/d1b.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00963226 s, 109 MB/s
prepared Wed May  8 18:05:31 EDT 2024.
fail_loc=0x198
fail_loc=0
stop mds1
stop mds2
removing OI files on mds1: idx=
removed '/mnt/lustre-brpt/oi.16.0'
removed '/mnt/lustre-brpt/oi.16.1'
removed '/mnt/lustre-brpt/oi.16.10'
removed '/mnt/lustre-brpt/oi.16.11'
removed '/mnt/lustre-brpt/oi.16.12'
removed '/mnt/lustre-brpt/oi.16.13'
removed '/mnt/lustre-brpt/oi.16.14'
removed '/mnt/lustre-brpt/oi.16.15'
removed '/mnt/lustre-brpt/oi.16.16'
removed '/mnt/lustre-brpt/oi.16.17'
removed '/mnt/lustre-brpt/oi.16.18'
removed '/mnt/lustre-brpt/oi.16.19'
removed '/mnt/lustre-brpt/oi.16.2'
removed '/mnt/lustre-brpt/oi.16.20'
removed '/mnt/lustre-brpt/oi.16.21'
removed '/mnt/lustre-brpt/oi.16.22'
removed '/mnt/lustre-brpt/oi.16.23'
removed '/mnt/lustre-brpt/oi.16.24'
removed '/mnt/lustre-brpt/oi.16.25'
removed '/mnt/lustre-brpt/oi.16.26'
removed '/mnt/lustre-brpt/oi.16.27'
removed '/mnt/lustre-brpt/oi.16.28'
removed '/mnt/lustre-brpt/oi.16.29'
removed '/mnt/lustre-brpt/oi.16.3'
removed '/mnt/lustre-brpt/oi.16.30'
removed '/mnt/lustre-brpt/oi.16.31'
removed '/mnt/lustre-brpt/oi.16.32'
removed '/mnt/lustre-brpt/oi.16.33'
removed '/mnt/lustre-brpt/oi.16.34'
removed '/mnt/lustre-brpt/oi.16.35'
removed '/mnt/lustre-brpt/oi.16.36'
removed '/mnt/lustre-brpt/oi.16.37'
removed '/mnt/lustre-brpt/oi.16.38'
removed '/mnt/lustre-brpt/oi.16.39'
removed '/mnt/lustre-brpt/oi.16.4'
removed '/mnt/lustre-brpt/oi.16.40'
removed '/mnt/lustre-brpt/oi.16.41'
removed '/mnt/lustre-brpt/oi.16.42'
removed '/mnt/lustre-brpt/oi.16.43'
removed '/mnt/lustre-brpt/oi.16.44'
removed '/mnt/lustre-brpt/oi.16.45'
removed '/mnt/lustre-brpt/oi.16.46'
removed '/mnt/lustre-brpt/oi.16.47'
removed '/mnt/lustre-brpt/oi.16.48'
removed '/mnt/lustre-brpt/oi.16.49'
removed '/mnt/lustre-brpt/oi.16.5'
removed '/mnt/lustre-brpt/oi.16.50'
removed '/mnt/lustre-brpt/oi.16.51'
removed '/mnt/lustre-brpt/oi.16.52'
removed '/mnt/lustre-brpt/oi.16.53'
removed '/mnt/lustre-brpt/oi.16.54'
removed '/mnt/lustre-brpt/oi.16.55'
removed '/mnt/lustre-brpt/oi.16.56'
removed '/mnt/lustre-brpt/oi.16.57'
removed '/mnt/lustre-brpt/oi.16.58'
removed '/mnt/lustre-brpt/oi.16.59'
removed '/mnt/lustre-brpt/oi.16.6'
removed '/mnt/lustre-brpt/oi.16.60'
removed '/mnt/lustre-brpt/oi.16.61'
removed '/mnt/lustre-brpt/oi.16.62'
removed '/mnt/lustre-brpt/oi.16.63'
removed '/mnt/lustre-brpt/oi.16.7'
removed '/mnt/lustre-brpt/oi.16.8'
removed '/mnt/lustre-brpt/oi.16.9'
removing OI files on mds2: idx=
removed '/mnt/lustre-brpt/oi.16.0'
removed '/mnt/lustre-brpt/oi.16.1'
removed '/mnt/lustre-brpt/oi.16.10'
removed '/mnt/lustre-brpt/oi.16.11'
removed '/mnt/lustre-brpt/oi.16.12'
removed '/mnt/lustre-brpt/oi.16.13'
removed '/mnt/lustre-brpt/oi.16.14'
removed '/mnt/lustre-brpt/oi.16.15'
removed '/mnt/lustre-brpt/oi.16.16'
removed '/mnt/lustre-brpt/oi.16.17'
removed '/mnt/lustre-brpt/oi.16.18'
removed '/mnt/lustre-brpt/oi.16.19'
removed '/mnt/lustre-brpt/oi.16.2'
removed '/mnt/lustre-brpt/oi.16.20'
removed '/mnt/lustre-brpt/oi.16.21'
removed '/mnt/lustre-brpt/oi.16.22'
removed '/mnt/lustre-brpt/oi.16.23'
removed '/mnt/lustre-brpt/oi.16.24'
removed '/mnt/lustre-brpt/oi.16.25'
removed '/mnt/lustre-brpt/oi.16.26'
removed '/mnt/lustre-brpt/oi.16.27'
removed '/mnt/lustre-brpt/oi.16.28'
removed '/mnt/lustre-brpt/oi.16.29'
removed '/mnt/lustre-brpt/oi.16.3'
removed '/mnt/lustre-brpt/oi.16.30'
removed '/mnt/lustre-brpt/oi.16.31'
removed '/mnt/lustre-brpt/oi.16.32'
removed '/mnt/lustre-brpt/oi.16.33'
removed '/mnt/lustre-brpt/oi.16.34'
removed '/mnt/lustre-brpt/oi.16.35'
removed '/mnt/lustre-brpt/oi.16.36'
removed '/mnt/lustre-brpt/oi.16.37'
removed '/mnt/lustre-brpt/oi.16.38'
removed '/mnt/lustre-brpt/oi.16.39'
removed '/mnt/lustre-brpt/oi.16.4'
removed '/mnt/lustre-brpt/oi.16.40'
removed '/mnt/lustre-brpt/oi.16.41'
removed '/mnt/lustre-brpt/oi.16.42'
removed '/mnt/lustre-brpt/oi.16.43'
removed '/mnt/lustre-brpt/oi.16.44'
removed '/mnt/lustre-brpt/oi.16.45'
removed '/mnt/lustre-brpt/oi.16.46'
removed '/mnt/lustre-brpt/oi.16.47'
removed '/mnt/lustre-brpt/oi.16.48'
removed '/mnt/lustre-brpt/oi.16.49'
removed '/mnt/lustre-brpt/oi.16.5'
removed '/mnt/lustre-brpt/oi.16.50'
removed '/mnt/lustre-brpt/oi.16.51'
removed '/mnt/lustre-brpt/oi.16.52'
removed '/mnt/lustre-brpt/oi.16.53'
removed '/mnt/lustre-brpt/oi.16.54'
removed '/mnt/lustre-brpt/oi.16.55'
removed '/mnt/lustre-brpt/oi.16.56'
removed '/mnt/lustre-brpt/oi.16.57'
removed '/mnt/lustre-brpt/oi.16.58'
removed '/mnt/lustre-brpt/oi.16.59'
removed '/mnt/lustre-brpt/oi.16.6'
removed '/mnt/lustre-brpt/oi.16.60'
removed '/mnt/lustre-brpt/oi.16.61'
removed '/mnt/lustre-brpt/oi.16.62'
removed '/mnt/lustre-brpt/oi.16.63'
removed '/mnt/lustre-brpt/oi.16.7'
removed '/mnt/lustre-brpt/oi.16.8'
removed '/mnt/lustre-brpt/oi.16.9'
start MDTs without disabling OI scrub
oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Starting client: oleg351-client.virtnet:  -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre
PASS 1b (49s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-scrub test 1c: Auto detect kinds of OI file(s) removed/recreated cases ========================================================== 18:06:01 (1715205961)
preparing... Wed May  8 18:06:02 EDT 2024
creating 0 files on mds1
striped dir -i0 -c1 -H all_char /mnt/lustre/d1c.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00913251 s, 115 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H fnv_1a_64 /mnt/lustre/d1c.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00968235 s, 108 MB/s
prepared Wed May  8 18:06:03 EDT 2024.
stop mds1
stop mds2
removing OI files on mds1: idx=0
removed '/mnt/lustre-brpt/oi.16.0'
removing OI files on mds2: idx=0
removed '/mnt/lustre-brpt/oi.16.0'
start MDTs with OI scrub disabled
oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Started LFSCK on the device lustre-MDT0000: scrub
Started LFSCK on the device lustre-MDT0001: scrub
Checking servers environments
Checking clients oleg351-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg351-server'
oleg351-server: oleg351-server.virtnet: executing load_modules_local
oleg351-server: Loading modules from /home/green/git/lustre-release/lustre
oleg351-server: detected 4 online CPUs by sysfs
oleg351-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg351-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg351-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg351-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg351-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg351-client.virtnet:  -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre
Starting client oleg351-client.virtnet:  -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre
Started clients oleg351-client.virtnet: 
192.168.203.151@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800b5e1c000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800b5e1c000.idle_timeout=debug
disable quota as required
preparing... Wed May  8 18:06:44 EDT 2024
creating 0 files on mds1
striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d1c.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0124955 s, 83.9 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H crush2 /mnt/lustre/d1c.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0105257 s, 99.6 MB/s
prepared Wed May  8 18:06:45 EDT 2024.
stop mds1
stop mds2
removing OI files on mds1: idx=2
removed '/mnt/lustre-brpt/oi.16.2'
removed '/mnt/lustre-brpt/oi.16.4'
removed '/mnt/lustre-brpt/oi.16.8'
removed '/mnt/lustre-brpt/oi.16.16'
removed '/mnt/lustre-brpt/oi.16.32'
removing OI files on mds2: idx=2
removed '/mnt/lustre-brpt/oi.16.2'
removed '/mnt/lustre-brpt/oi.16.4'
removed '/mnt/lustre-brpt/oi.16.8'
removed '/mnt/lustre-brpt/oi.16.16'
removed '/mnt/lustre-brpt/oi.16.32'
start MDTs with OI scrub disabled
oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Started LFSCK on the device lustre-MDT0000: scrub
Started LFSCK on the device lustre-MDT0001: scrub
Checking servers environments
Checking clients oleg351-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
loading modules on: 'oleg351-server'
oleg351-server: oleg351-server.virtnet: executing load_modules_local
oleg351-server: Loading modules from /home/green/git/lustre-release/lustre
oleg351-server: detected 4 online CPUs by sysfs
oleg351-server: Force libcfs to create 2 CPU partitions
oleg351-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg351-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg351-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg351-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg351-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg351-client.virtnet:  -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre
Starting client oleg351-client.virtnet:  -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre
Started clients oleg351-client.virtnet: 
192.168.203.151@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800b62b2000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800b62b2000.idle_timeout=debug
disable quota as required
preparing... Wed May  8 18:07:26 EDT 2024
creating 0 files on mds1
striped dir -i0 -c1 -H crush /mnt/lustre/d1c.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00931763 s, 113 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H all_char /mnt/lustre/d1c.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0100578 s, 104 MB/s
prepared Wed May  8 18:07:28 EDT 2024.
stop mds1
stop mds2
removing OI files on mds1: idx=3
removed '/mnt/lustre-brpt/oi.16.3'
removed '/mnt/lustre-brpt/oi.16.9'
removed '/mnt/lustre-brpt/oi.16.27'
removing OI files on mds2: idx=3
removed '/mnt/lustre-brpt/oi.16.3'
removed '/mnt/lustre-brpt/oi.16.9'
removed '/mnt/lustre-brpt/oi.16.27'
start MDTs with OI scrub disabled
oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Started LFSCK on the device lustre-MDT0000: scrub
Started LFSCK on the device lustre-MDT0001: scrub
PASS 1c (109s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-scrub test 2: Trigger OI scrub when MDT mounts for backup/restore case ========================================================== 18:07:52 (1715206072)
Checking servers environments
Checking clients oleg351-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg351-server'
oleg351-server: oleg351-server.virtnet: executing load_modules_local
oleg351-server: Loading modules from /home/green/git/lustre-release/lustre
oleg351-server: detected 4 online CPUs by sysfs
oleg351-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg351-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg351-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg351-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg351-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg351-client.virtnet:  -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre
Starting client oleg351-client.virtnet:  -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre
Started clients oleg351-client.virtnet: 
192.168.203.151@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff880137262000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff880137262000.idle_timeout=debug
disable quota as required
preparing... Wed May  8 18:08:11 EDT 2024
creating 0 files on mds1
striped dir -i0 -c1 -H crush2 /mnt/lustre/d2.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00772907 s, 136 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H crush2 /mnt/lustre/d2.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00834536 s, 126 MB/s
prepared Wed May  8 18:08:12 EDT 2024.
stop mds1
stop mds2
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
starting MDTs without disabling OI scrub
oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Starting client: oleg351-client.virtnet:  -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre
PASS 2 (57s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-scrub test 4a: Auto trigger OI scrub if bad OI mapping was found (1) ========================================================== 18:08:50 (1715206130)
preparing... Wed May  8 18:08:51 EDT 2024
creating 0 files on mds1
striped dir -i0 -c1 -H crush /mnt/lustre/d4a.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0102642 s, 102 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H crush2 /mnt/lustre/d4a.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00965566 s, 109 MB/s
prepared Wed May  8 18:08:53 EDT 2024.
stop mds1
stop mds2
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
starting MDTs with OI scrub disabled
oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Starting client: oleg351-client.virtnet:  -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre
PASS 4a (55s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-scrub test 4b: Auto trigger OI scrub if bad OI mapping was found (2) ========================================================== 18:09:46 (1715206186)
preparing... Wed May  8 18:09:47 EDT 2024
creating 5 files on mds1
striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d4b.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00851584 s, 123 MB/s
creating 5 files on mds2
striped dir -i1 -c1 -H fnv_1a_64 /mnt/lustre/d4b.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00961992 s, 109 MB/s
prepared Wed May  8 18:09:48 EDT 2024.
stop mds1
stop mds2
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
starting MDTs with OI scrub disabled
oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Starting client: oleg351-client.virtnet:  -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre
OI scrub on MDS1 status for the 1st time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 65 seconds
time_since_latest_start: 8 seconds
time_since_last_checkpoint: 8 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 3
updated: 0
failed: 0
prior_updated: 3
noscrub: 0
igif: 0
success_count: 6
run_time: 8 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS2 status for the 1st time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 64 seconds
time_since_latest_start: 5 seconds
time_since_last_checkpoint: 5 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 1
updated: 0
failed: 0
prior_updated: 1
noscrub: 0
igif: 0
success_count: 6
run_time: 6 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS1 status for the 2nd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 72 seconds
time_since_latest_start: 15 seconds
time_since_last_checkpoint: 15 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 4
updated: 0
failed: 0
prior_updated: 4
noscrub: 0
igif: 0
success_count: 6
run_time: 15 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS2 status for the 2nd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 71 seconds
time_since_latest_start: 12 seconds
time_since_last_checkpoint: 12 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 2
updated: 0
failed: 0
prior_updated: 2
noscrub: 0
igif: 0
success_count: 6
run_time: 12 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS1 status for the 3rd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 39 seconds
time_since_latest_start: 39 seconds
time_since_last_checkpoint: 39 seconds
latest_start_position: 12
last_checkpoint_position: 160001
first_failure_position: N/A
checked: 259
updated: 50
failed: 0
prior_updated: 38
noscrub: 70
igif: 1
success_count: 7
run_time: 0 seconds
average_speed: 259 objects/sec
real_time_speed: N/A
current_position: N/A
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS2 status for the 3rd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 6 seconds
time_since_latest_start: 6 seconds
time_since_last_checkpoint: 6 seconds
latest_start_position: 12
last_checkpoint_position: 160001
first_failure_position: N/A
checked: 240
updated: 51
failed: 0
prior_updated: 36
noscrub: 73
igif: 1
success_count: 7
run_time: 0 seconds
average_speed: 240 objects/sec
real_time_speed: N/A
current_position: N/A
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
total 1024
-rw-r--r-- 1 root root       0 May  8 18:09 f1
-rw-r--r-- 1 root root 1048576 May  8 18:09 f2
total 1024
-rw-r--r-- 1 root root       0 May  8 18:09 f1
-rw-r--r-- 1 root root 1048576 May  8 18:09 f2
PASS 4b (137s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-scrub test 4c: Auto trigger OI scrub if bad OI mapping was found (3) ========================================================== 18:12:04 (1715206324)
preparing... Wed May  8 18:12:05 EDT 2024
creating 500 files on mds1
striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d4c.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00822159 s, 128 MB/s
creating 500 files on mds2
striped dir -i1 -c1 -H all_char /mnt/lustre/d4c.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00811573 s, 129 MB/s
prepared Wed May  8 18:12:09 EDT 2024.
stop mds1
stop mds2
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
starting MDTs with OI scrub disabled
oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Starting client: oleg351-client.virtnet:  -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre
OI scrub on MDS1 status for the 1st time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 95 seconds
time_since_latest_start: 8 seconds
time_since_last_checkpoint: 8 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 3
updated: 0
failed: 0
prior_updated: 3
noscrub: 0
igif: 0
success_count: 7
run_time: 8 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS2 status for the 1st time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 62 seconds
time_since_latest_start: 6 seconds
time_since_last_checkpoint: 6 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 1
updated: 0
failed: 0
prior_updated: 1
noscrub: 0
igif: 0
success_count: 7
run_time: 6 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS1 status for the 2nd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 102 seconds
time_since_latest_start: 15 seconds
time_since_last_checkpoint: 15 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 4
updated: 0
failed: 0
prior_updated: 4
noscrub: 0
igif: 0
success_count: 7
run_time: 15 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS2 status for the 2nd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 69 seconds
time_since_latest_start: 13 seconds
time_since_last_checkpoint: 13 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 2
updated: 0
failed: 0
prior_updated: 2
noscrub: 0
igif: 0
success_count: 7
run_time: 13 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS1 status for the 3rd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 24 seconds
time_since_latest_start: 24 seconds
time_since_last_checkpoint: 24 seconds
latest_start_position: 12
last_checkpoint_position: 160001
first_failure_position: N/A
checked: 756
updated: 562
failed: 0
prior_updated: 21
noscrub: 70
igif: 1
success_count: 8
run_time: 0 seconds
average_speed: 756 objects/sec
real_time_speed: N/A
current_position: N/A
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS2 status for the 3rd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 6 seconds
time_since_latest_start: 6 seconds
time_since_last_checkpoint: 6 seconds
latest_start_position: 12
last_checkpoint_position: 160001
first_failure_position: N/A
checked: 737
updated: 561
failed: 0
prior_updated: 21
noscrub: 73
igif: 1
success_count: 8
run_time: 0 seconds
average_speed: 737 objects/sec
real_time_speed: N/A
current_position: N/A
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
-r--r--r-- 1 root root 0 May  8 18:12 /mnt/lustre/d4c.sanity-scrub/mds1/f4c.sanity-scrub1
-r--r--r-- 1 root root 0 May  8 18:12 /mnt/lustre/d4c.sanity-scrub/mds2/f4c.sanity-scrub1
PASS 4c (103s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-scrub test 4d: FID in LMA mismatch with object FID won't block create ========================================================== 18:13:48 (1715206428)
fail_loc=0x19b
fail_loc=0
PASS 4d (6s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-scrub test 4e: FID reuse can be fixed ========== 18:13:56 (1715206436)
fail_loc=0x1a0
fail_loc=0
Started LFSCK on the device lustre-OST0000: scrub
PASS 4e (3s)
debug_raw_pointers=0
debug_raw_pointers=0
debug_raw_pointers=Y
debug_raw_pointers=Y

== sanity-scrub test 5: OI scrub state machine =========== 18:14:01 (1715206441)