-----============= acceptance-small: sanity-scrub ============----- Wed Nov 13 12:08:46 EST 2024
mgs: CentOS Linux release 7.9.2009 (Core)
MGS_OS_VERSION_ID=7
MGS_OS_ID=centos
MGS_OS_VERSION_CODE=117440512
MGS_OS_ID_LIKE=rhel fedora centos
mds1: CentOS Linux release 7.9.2009 (Core)
MDS1_OS_ID_LIKE=rhel fedora centos
MDS1_OS_ID=centos
MDS1_OS_VERSION_ID=7
MDS1_OS_VERSION_CODE=117440512
ost1: CentOS Linux release 7.9.2009 (Core)
OST1_OS_VERSION_CODE=117440512
OST1_OS_VERSION_ID=7
OST1_OS_ID_LIKE=rhel fedora centos
OST1_OS_ID=centos
client: CentOS Linux release 7.9.2009 (Core)
CLIENT_OS_ID=centos
CLIENT_OS_ID_LIKE=rhel fedora centos
CLIENT_OS_VERSION_ID=7
CLIENT_OS_VERSION_CODE=117440512
excepting tests: 
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg415-server'
oleg415-server: oleg415-server.virtnet: executing load_modules_local
oleg415-server: Loading modules from /home/green/git/lustre-release/lustre
oleg415-server: detected 4 online CPUs by sysfs
oleg415-server: Force libcfs to create 2 CPU partitions
client=34603009 MDS=34603009 OSS=34603009
Stopping clients: oleg415-client.virtnet /mnt/lustre (opts:)
Stopping client oleg415-client.virtnet /mnt/lustre opts:
Stopping clients: oleg415-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg415-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg415-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg415-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg415-server
unloading modules via unload_modules_local on: 'oleg415-server'
oleg415-server: oleg415-server.virtnet: executing unload_modules_local
modules unloaded.
=== sanity-scrub: start setup 12:09:31 (1731517771) ===
Stopping clients: oleg415-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg415-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 2
oleg415-server: oleg415-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg415-server'
oleg415-server: oleg415-server.virtnet: executing load_modules_local
oleg415-server: Loading modules from /home/green/git/lustre-release/lustre
oleg415-server: detected 4 online CPUs by sysfs
oleg415-server: Force libcfs to create 2 CPU partitions
oleg415-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg415-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format mds1: /dev/vdc
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Format mds2: /dev/vdd
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Format ost1: /dev/vde
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Format ost2: /dev/vdf
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Checking servers environments
Checking clients oleg415-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
loading modules on: 'oleg415-server'
oleg415-server: oleg415-server.virtnet: executing load_modules_local
oleg415-server: Loading modules from /home/green/git/lustre-release/lustre
oleg415-server: detected 4 online CPUs by sysfs
oleg415-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Commit the device label on /dev/vdc
Started lustre-MDT0000
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Commit the device label on /dev/vdd
Started lustre-MDT0001
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Commit the device label on /dev/vde
Started lustre-OST0000
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Commit the device label on /dev/vdf
Started lustre-OST0001
Starting client: oleg415-client.virtnet:  -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre
Starting client oleg415-client.virtnet:  -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre
Started clients oleg415-client.virtnet: 
192.168.204.115@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800a9525800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800a9525800.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
Updated after 5s: want 'procname_uid' got 'procname_uid'
disable quota as required
osd-ldiskfs.track_declares_assert=1
=== sanity-scrub: finish setup 12:10:33 (1731517833) ===

== sanity-scrub test 0: Do not auto trigger OI scrub for non-backup/restore case ========================================================== 12:10:33 (1731517833)
preparing... Wed Nov 13 12:10:34 EST 2024
creating 0 files on mds1
striped dir -i0 -c1 -H all_char /mnt/lustre/d0.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00849744 s, 123 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H crush2 /mnt/lustre/d0.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0113 s, 92.8 MB/s
prepared Wed Nov 13 12:10:35 EST 2024.
stop mds1
stop mds2
starting MDTs without disabling OI scrub
oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Starting client: oleg415-client.virtnet:  -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre
PASS 0 (29s)

== sanity-scrub test 1a: Auto trigger initial OI scrub when server mounts ========================================================== 12:11:02 (1731517862)
preparing... Wed Nov 13 12:11:02 EST 2024
creating 0 files on mds1
striped dir -i0 -c1 -H all_char /mnt/lustre/d1a.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0129712 s, 80.8 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H crush /mnt/lustre/d1a.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0092979 s, 113 MB/s
prepared Wed Nov 13 12:11:04 EST 2024.
stop mds1
stop mds2
start mds1 without disabling OI scrub: -o localrecov -o user_xattr
oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Starting client: oleg415-client.virtnet:  -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre
fail_loc=0x193
fail_loc=0
192.168.204.115@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0
Stopping client oleg415-client.virtnet /mnt/lustre (opts:)
stop mds1
start mds1 with disabling OI scrub: -o localrecov -o user_xattr,noscrub
oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
PASS 1a (33s)

== sanity-scrub test 1b: Trigger OI scrub when MDT mounts for OI files remove/recreate case ========================================================== 12:11:35 (1731517895)
Checking servers environments
Checking clients oleg415-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg415-server'
oleg415-server: oleg415-server.virtnet: executing load_modules_local
oleg415-server: Loading modules from /home/green/git/lustre-release/lustre
oleg415-server: detected 4 online CPUs by sysfs
oleg415-server: Force libcfs to create 2 CPU partitions
oleg415-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg415-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg415-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg415-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg415-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg415-client.virtnet:  -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre
Starting client oleg415-client.virtnet:  -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre
Started clients oleg415-client.virtnet: 
192.168.204.115@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800a8e22800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800a8e22800.idle_timeout=debug
disable quota as required
preparing... Wed Nov 13 12:11:55 EST 2024
creating 0 files on mds1
striped dir -i0 -c1 -H crush /mnt/lustre/d1b.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0162543 s, 64.5 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H crush /mnt/lustre/d1b.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0142109 s, 73.8 MB/s
prepared Wed Nov 13 12:11:57 EST 2024.
fail_loc=0x198
fail_loc=0
stop mds1
stop mds2
removing OI files on mds1: idx=
removed '/mnt/lustre-brpt/oi.16.0'
removed '/mnt/lustre-brpt/oi.16.1'
removed '/mnt/lustre-brpt/oi.16.10'
removed '/mnt/lustre-brpt/oi.16.11'
removed '/mnt/lustre-brpt/oi.16.12'
removed '/mnt/lustre-brpt/oi.16.13'
removed '/mnt/lustre-brpt/oi.16.14'
removed '/mnt/lustre-brpt/oi.16.15'
removed '/mnt/lustre-brpt/oi.16.16'
removed '/mnt/lustre-brpt/oi.16.17'
removed '/mnt/lustre-brpt/oi.16.18'
removed '/mnt/lustre-brpt/oi.16.19'
removed '/mnt/lustre-brpt/oi.16.2'
removed '/mnt/lustre-brpt/oi.16.20'
removed '/mnt/lustre-brpt/oi.16.21'
removed '/mnt/lustre-brpt/oi.16.22'
removed '/mnt/lustre-brpt/oi.16.23'
removed '/mnt/lustre-brpt/oi.16.24'
removed '/mnt/lustre-brpt/oi.16.25'
removed '/mnt/lustre-brpt/oi.16.26'
removed '/mnt/lustre-brpt/oi.16.27'
removed '/mnt/lustre-brpt/oi.16.28'
removed '/mnt/lustre-brpt/oi.16.29'
removed '/mnt/lustre-brpt/oi.16.3'
removed '/mnt/lustre-brpt/oi.16.30'
removed '/mnt/lustre-brpt/oi.16.31'
removed '/mnt/lustre-brpt/oi.16.32'
removed '/mnt/lustre-brpt/oi.16.33'
removed '/mnt/lustre-brpt/oi.16.34'
removed '/mnt/lustre-brpt/oi.16.35'
removed '/mnt/lustre-brpt/oi.16.36'
removed '/mnt/lustre-brpt/oi.16.37'
removed '/mnt/lustre-brpt/oi.16.38'
removed '/mnt/lustre-brpt/oi.16.39'
removed '/mnt/lustre-brpt/oi.16.4'
removed '/mnt/lustre-brpt/oi.16.40'
removed '/mnt/lustre-brpt/oi.16.41'
removed '/mnt/lustre-brpt/oi.16.42'
removed '/mnt/lustre-brpt/oi.16.43'
removed '/mnt/lustre-brpt/oi.16.44'
removed '/mnt/lustre-brpt/oi.16.45'
removed '/mnt/lustre-brpt/oi.16.46'
removed '/mnt/lustre-brpt/oi.16.47'
removed '/mnt/lustre-brpt/oi.16.48'
removed '/mnt/lustre-brpt/oi.16.49'
removed '/mnt/lustre-brpt/oi.16.5'
removed '/mnt/lustre-brpt/oi.16.50'
removed '/mnt/lustre-brpt/oi.16.51'
removed '/mnt/lustre-brpt/oi.16.52'
removed '/mnt/lustre-brpt/oi.16.53'
removed '/mnt/lustre-brpt/oi.16.54'
removed '/mnt/lustre-brpt/oi.16.55'
removed '/mnt/lustre-brpt/oi.16.56'
removed '/mnt/lustre-brpt/oi.16.57'
removed '/mnt/lustre-brpt/oi.16.58'
removed '/mnt/lustre-brpt/oi.16.59'
removed '/mnt/lustre-brpt/oi.16.6'
removed '/mnt/lustre-brpt/oi.16.60'
removed '/mnt/lustre-brpt/oi.16.61'
removed '/mnt/lustre-brpt/oi.16.62'
removed '/mnt/lustre-brpt/oi.16.63'
removed '/mnt/lustre-brpt/oi.16.7'
removed '/mnt/lustre-brpt/oi.16.8'
removed '/mnt/lustre-brpt/oi.16.9'
removing OI files on mds2: idx=
removed '/mnt/lustre-brpt/oi.16.0'
removed '/mnt/lustre-brpt/oi.16.1'
removed '/mnt/lustre-brpt/oi.16.10'
removed '/mnt/lustre-brpt/oi.16.11'
removed '/mnt/lustre-brpt/oi.16.12'
removed '/mnt/lustre-brpt/oi.16.13'
removed '/mnt/lustre-brpt/oi.16.14'
removed '/mnt/lustre-brpt/oi.16.15'
removed '/mnt/lustre-brpt/oi.16.16'
removed '/mnt/lustre-brpt/oi.16.17'
removed '/mnt/lustre-brpt/oi.16.18'
removed '/mnt/lustre-brpt/oi.16.19'
removed '/mnt/lustre-brpt/oi.16.2'
removed '/mnt/lustre-brpt/oi.16.20'
removed '/mnt/lustre-brpt/oi.16.21'
removed '/mnt/lustre-brpt/oi.16.22'
removed '/mnt/lustre-brpt/oi.16.23'
removed '/mnt/lustre-brpt/oi.16.24'
removed '/mnt/lustre-brpt/oi.16.25'
removed '/mnt/lustre-brpt/oi.16.26'
removed '/mnt/lustre-brpt/oi.16.27'
removed '/mnt/lustre-brpt/oi.16.28'
removed '/mnt/lustre-brpt/oi.16.29'
removed '/mnt/lustre-brpt/oi.16.3'
removed '/mnt/lustre-brpt/oi.16.30'
removed '/mnt/lustre-brpt/oi.16.31'
removed '/mnt/lustre-brpt/oi.16.32'
removed '/mnt/lustre-brpt/oi.16.33'
removed '/mnt/lustre-brpt/oi.16.34'
removed '/mnt/lustre-brpt/oi.16.35'
removed '/mnt/lustre-brpt/oi.16.36'
removed '/mnt/lustre-brpt/oi.16.37'
removed '/mnt/lustre-brpt/oi.16.38'
removed '/mnt/lustre-brpt/oi.16.39'
removed '/mnt/lustre-brpt/oi.16.4'
removed '/mnt/lustre-brpt/oi.16.40'
removed '/mnt/lustre-brpt/oi.16.41'
removed '/mnt/lustre-brpt/oi.16.42'
removed '/mnt/lustre-brpt/oi.16.43'
removed '/mnt/lustre-brpt/oi.16.44'
removed '/mnt/lustre-brpt/oi.16.45'
removed '/mnt/lustre-brpt/oi.16.46'
removed '/mnt/lustre-brpt/oi.16.47'
removed '/mnt/lustre-brpt/oi.16.48'
removed '/mnt/lustre-brpt/oi.16.49'
removed '/mnt/lustre-brpt/oi.16.5'
removed '/mnt/lustre-brpt/oi.16.50'
removed '/mnt/lustre-brpt/oi.16.51'
removed '/mnt/lustre-brpt/oi.16.52'
removed '/mnt/lustre-brpt/oi.16.53'
removed '/mnt/lustre-brpt/oi.16.54'
removed '/mnt/lustre-brpt/oi.16.55'
removed '/mnt/lustre-brpt/oi.16.56'
removed '/mnt/lustre-brpt/oi.16.57'
removed '/mnt/lustre-brpt/oi.16.58'
removed '/mnt/lustre-brpt/oi.16.59'
removed '/mnt/lustre-brpt/oi.16.6'
removed '/mnt/lustre-brpt/oi.16.60'
removed '/mnt/lustre-brpt/oi.16.61'
removed '/mnt/lustre-brpt/oi.16.62'
removed '/mnt/lustre-brpt/oi.16.63'
removed '/mnt/lustre-brpt/oi.16.7'
removed '/mnt/lustre-brpt/oi.16.8'
removed '/mnt/lustre-brpt/oi.16.9'
start MDTs without disabling OI scrub
oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Starting client: oleg415-client.virtnet:  -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre
PASS 1b (50s)

== sanity-scrub test 1c: Auto detect kinds of OI file(s) removed/recreated cases ========================================================== 12:12:25 (1731517945)
preparing... Wed Nov 13 12:12:26 EST 2024
creating 0 files on mds1
striped dir -i0 -c1 -H crush /mnt/lustre/d1c.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00903913 s, 116 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H all_char /mnt/lustre/d1c.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0109859 s, 95.4 MB/s
prepared Wed Nov 13 12:12:27 EST 2024.
stop mds1
stop mds2
removing OI files on mds1: idx=0
removed '/mnt/lustre-brpt/oi.16.0'
removing OI files on mds2: idx=0
removed '/mnt/lustre-brpt/oi.16.0'
start MDTs with OI scrub disabled
oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Started LFSCK on the device lustre-MDT0000: scrub
Started LFSCK on the device lustre-MDT0001: scrub
Checking servers environments
Checking clients oleg415-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg415-server'
oleg415-server: oleg415-server.virtnet: executing load_modules_local
oleg415-server: Loading modules from /home/green/git/lustre-release/lustre
oleg415-server: detected 4 online CPUs by sysfs
oleg415-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg415-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg415-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg415-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg415-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg415-client.virtnet:  -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre
Starting client oleg415-client.virtnet:  -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre
Started clients oleg415-client.virtnet: 
192.168.204.115@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800b596d800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800b596d800.idle_timeout=debug
disable quota as required
preparing... Wed Nov 13 12:13:08 EST 2024
creating 0 files on mds1
striped dir -i0 -c1 -H all_char /mnt/lustre/d1c.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00908032 s, 115 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H crush2 /mnt/lustre/d1c.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00786939 s, 133 MB/s
prepared Wed Nov 13 12:13:10 EST 2024.
stop mds1
stop mds2
removing OI files on mds1: idx=2
removed '/mnt/lustre-brpt/oi.16.2'
removed '/mnt/lustre-brpt/oi.16.4'
removed '/mnt/lustre-brpt/oi.16.8'
removed '/mnt/lustre-brpt/oi.16.16'
removed '/mnt/lustre-brpt/oi.16.32'
removing OI files on mds2: idx=2
removed '/mnt/lustre-brpt/oi.16.2'
removed '/mnt/lustre-brpt/oi.16.4'
removed '/mnt/lustre-brpt/oi.16.8'
removed '/mnt/lustre-brpt/oi.16.16'
removed '/mnt/lustre-brpt/oi.16.32'
start MDTs with OI scrub disabled
oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Started LFSCK on the device lustre-MDT0000: scrub
Started LFSCK on the device lustre-MDT0001: scrub
Checking servers environments
Checking clients oleg415-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg415-server'
oleg415-server: oleg415-server.virtnet: executing load_modules_local
oleg415-server: Loading modules from /home/green/git/lustre-release/lustre
oleg415-server: detected 4 online CPUs by sysfs
oleg415-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg415-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg415-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg415-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg415-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg415-client.virtnet:  -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre
Starting client oleg415-client.virtnet:  -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre
Started clients oleg415-client.virtnet: 
192.168.204.115@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800b59a2000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800b59a2000.idle_timeout=debug
disable quota as required
preparing... Wed Nov 13 12:13:50 EST 2024
creating 0 files on mds1
striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d1c.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0113045 s, 92.8 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H crush2 /mnt/lustre/d1c.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0116143 s, 90.3 MB/s
prepared Wed Nov 13 12:13:52 EST 2024.
stop mds1
stop mds2
removing OI files on mds1: idx=3
removed '/mnt/lustre-brpt/oi.16.3'
removed '/mnt/lustre-brpt/oi.16.9'
removed '/mnt/lustre-brpt/oi.16.27'
removing OI files on mds2: idx=3
removed '/mnt/lustre-brpt/oi.16.3'
removed '/mnt/lustre-brpt/oi.16.9'
removed '/mnt/lustre-brpt/oi.16.27'
start MDTs with OI scrub disabled
oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Started LFSCK on the device lustre-MDT0000: scrub
Started LFSCK on the device lustre-MDT0001: scrub
PASS 1c (110s)

== sanity-scrub test 2: Trigger OI scrub when MDT mounts for backup/restore case ========================================================== 12:14:15 (1731518055)
Checking servers environments
Checking clients oleg415-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg415-server'
oleg415-server: oleg415-server.virtnet: executing load_modules_local
oleg415-server: Loading modules from /home/green/git/lustre-release/lustre
oleg415-server: detected 4 online CPUs by sysfs
oleg415-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg415-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Starting mds2: -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg415-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg415-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg415-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg415-client.virtnet:  -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre
Starting client oleg415-client.virtnet:  -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre
Started clients oleg415-client.virtnet: 
192.168.204.115@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8800a95f5000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8800a95f5000.idle_timeout=debug
disable quota as required
preparing... Wed Nov 13 12:14:34 EST 2024
creating 0 files on mds1
striped dir -i0 -c1 -H crush /mnt/lustre/d2.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00995032 s, 105 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H all_char /mnt/lustre/d2.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00978905 s, 107 MB/s
prepared Wed Nov 13 12:14:35 EST 2024.
stop mds1
stop mds2
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
starting MDTs without disabling OI scrub
oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Starting client: oleg415-client.virtnet:  -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre
PASS 2 (61s)

== sanity-scrub test 4a: Auto trigger OI scrub if bad OI mapping was found (1) ========================================================== 12:15:16 (1731518116)
preparing... Wed Nov 13 12:15:17 EST 2024
creating 0 files on mds1
striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d4a.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00875641 s, 120 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H crush /mnt/lustre/d4a.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0106276 s, 98.7 MB/s
prepared Wed Nov 13 12:15:19 EST 2024.
stop mds1
stop mds2
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
starting MDTs with OI scrub disabled
oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Starting client: oleg415-client.virtnet:  -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre
PASS 4a (55s)

== sanity-scrub test 4b: Auto trigger OI scrub if bad OI mapping was found (2) ========================================================== 12:16:11 (1731518171)
preparing... Wed Nov 13 12:16:11 EST 2024
creating 5 files on mds1
striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d4b.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00917774 s, 114 MB/s
creating 5 files on mds2
striped dir -i1 -c1 -H all_char /mnt/lustre/d4b.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00933575 s, 112 MB/s
prepared Wed Nov 13 12:16:13 EST 2024.
stop mds1
stop mds2
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
starting MDTs with OI scrub disabled
oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Starting client: oleg415-client.virtnet:  -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre
OI scrub on MDS1 status for the 1st time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 58 seconds
time_since_latest_start: 7 seconds
time_since_last_checkpoint: 7 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 3
updated: 0
failed: 0
prior_updated: 3
noscrub: 0
igif: 0
success_count: 6
run_time: 7 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS2 status for the 1st time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 58 seconds
time_since_latest_start: 6 seconds
time_since_last_checkpoint: 6 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 1
updated: 0
failed: 0
prior_updated: 1
noscrub: 0
igif: 0
success_count: 6
run_time: 6 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS1 status for the 2nd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 65 seconds
time_since_latest_start: 14 seconds
time_since_last_checkpoint: 14 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 4
updated: 0
failed: 0
prior_updated: 4
noscrub: 0
igif: 0
success_count: 6
run_time: 14 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS2 status for the 2nd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 64 seconds
time_since_latest_start: 12 seconds
time_since_last_checkpoint: 12 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 2
updated: 0
failed: 0
prior_updated: 2
noscrub: 0
igif: 0
success_count: 6
run_time: 13 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS1 status for the 3rd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 38 seconds
time_since_latest_start: 38 seconds
time_since_last_checkpoint: 38 seconds
latest_start_position: 12
last_checkpoint_position: 160001
first_failure_position: N/A
checked: 259
updated: 50
failed: 0
prior_updated: 38
noscrub: 70
igif: 1
success_count: 7
run_time: 0 seconds
average_speed: 259 objects/sec
real_time_speed: N/A
current_position: N/A
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS2 status for the 3rd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 6 seconds
time_since_latest_start: 6 seconds
time_since_last_checkpoint: 6 seconds
latest_start_position: 12
last_checkpoint_position: 160001
first_failure_position: N/A
checked: 237
updated: 51
failed: 0
prior_updated: 36
noscrub: 73
igif: 1
success_count: 7
run_time: 0 seconds
average_speed: 237 objects/sec
real_time_speed: N/A
current_position: N/A
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
total 1024
-rw-r--r-- 1 root root       0 Nov 13 12:16 f1
-rw-r--r-- 1 root root 1048576 Nov 13 12:16 f2
total 1024
-rw-r--r-- 1 root root       0 Nov 13 12:16 f1
-rw-r--r-- 1 root root 1048576 Nov 13 12:16 f2
PASS 4b (131s)

== sanity-scrub test 4c: Auto trigger OI scrub if bad OI mapping was found (3) ========================================================== 12:18:22 (1731518302)
preparing... Wed Nov 13 12:18:22 EST 2024
creating 500 files on mds1
striped dir -i0 -c1 -H all_char /mnt/lustre/d4c.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0112677 s, 93.1 MB/s
creating 500 files on mds2
striped dir -i1 -c1 -H all_char /mnt/lustre/d4c.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00918098 s, 114 MB/s
prepared Wed Nov 13 12:18:26 EST 2024.
stop mds1
stop mds2
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
starting MDTs with OI scrub disabled
oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Starting client: oleg415-client.virtnet:  -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre
OI scrub on MDS1 status for the 1st time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 97 seconds
time_since_latest_start: 7 seconds
time_since_last_checkpoint: 7 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 3
updated: 0
failed: 0
prior_updated: 3
noscrub: 0
igif: 0
success_count: 7
run_time: 8 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS2 status for the 1st time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 65 seconds
time_since_latest_start: 6 seconds
time_since_last_checkpoint: 6 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 1
updated: 0
failed: 0
prior_updated: 1
noscrub: 0
igif: 0
success_count: 7
run_time: 6 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS1 status for the 2nd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 105 seconds
time_since_latest_start: 15 seconds
time_since_last_checkpoint: 15 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 4
updated: 0
failed: 0
prior_updated: 4
noscrub: 0
igif: 0
success_count: 7
run_time: 15 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS2 status for the 2nd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 73 seconds
time_since_latest_start: 14 seconds
time_since_last_checkpoint: 14 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 2
updated: 0
failed: 0
prior_updated: 2
noscrub: 0
igif: 0
success_count: 7
run_time: 14 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS1 status for the 3rd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 24 seconds
time_since_latest_start: 24 seconds
time_since_last_checkpoint: 24 seconds
latest_start_position: 12
last_checkpoint_position: 160001
first_failure_position: N/A
checked: 756
updated: 562
failed: 0
prior_updated: 21
noscrub: 70
igif: 1
success_count: 8
run_time: 0 seconds
average_speed: 756 objects/sec
real_time_speed: N/A
current_position: N/A
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS2 status for the 3rd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 6 seconds
time_since_latest_start: 6 seconds
time_since_last_checkpoint: 6 seconds
latest_start_position: 12
last_checkpoint_position: 160001
first_failure_position: N/A
checked: 734
updated: 561
failed: 0
prior_updated: 21
noscrub: 73
igif: 1
success_count: 8
run_time: 0 seconds
average_speed: 734 objects/sec
real_time_speed: N/A
current_position: N/A
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
-r--r--r-- 1 root root 0 Nov 13 12:18 /mnt/lustre/d4c.sanity-scrub/mds1/f4c.sanity-scrub1
-r--r--r-- 1 root root 0 Nov 13 12:18 /mnt/lustre/d4c.sanity-scrub/mds2/f4c.sanity-scrub1
PASS 4c (108s)

== sanity-scrub test 4d: FID in LMA mismatch with object FID won't block create ========================================================== 12:20:10 (1731518410)
fail_loc=0x19b
fail_loc=0
PASS 4d (6s)

== sanity-scrub test 4e: FID reuse can be fixed ========== 12:20:16 (1731518416)
 - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 801 prealloc_next_id: 774 count: 28
fail_loc=0x1a0
fail_loc=0
Started LFSCK on the device lustre-OST0000: scrub
PASS 4e (4s)

== sanity-scrub test 5: OI scrub state machine =========== 12:20:20 (1731518420)