-----============= acceptance-small: sanity-scrub ============----- Fri Sep  4 07:08:29 EDT 2026
MGS_OS_ID_LIKE=rhel centos fedora rocky
MGS_OS_VERSION_ID=8.10
MGS_OS_ID=rocky
MGS_OS_VERSION_CODE=134873088
MDS1_OS_VERSION_ID=8.10
MDS1_OS_VERSION_CODE=134873088
MDS1_OS_ID_LIKE=rhel centos fedora rocky
MDS1_OS_ID=rocky
OST1_OS_VERSION_CODE=134873088
OST1_OS_ID_LIKE=rhel centos fedora rocky
OST1_OS_VERSION_ID=8.10
OST1_OS_ID=rocky
CLIENT_OS_ID=rocky
CLIENT_OS_VERSION_CODE=134873088
CLIENT_OS_VERSION_ID=8.10
CLIENT_OS_ID_LIKE=rhel centos fedora rocky
oleg127-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-scrub.*ex': No such file or directory
excepting tests: 
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg127-server'
oleg127-server: oleg127-server.virtnet: executing load_modules_local
oleg127-server: kmemleak disabled
oleg127-server: Loading modules from /home/green/git/lustre-release/lustre
oleg127-server: detected 4 online CPUs by sysfs
oleg127-server: MODOPTS_LIBCFS=
oleg127-server: Force libcfs to create 2 CPU partitions
oleg127-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
client=34683394 MDS=34683394 OSS=34683394
Stopping clients: oleg127-client.virtnet /mnt/lustre (opts:)
Stopping client oleg127-client.virtnet /mnt/lustre opts:
Stopping clients: oleg127-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg127-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg127-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg127-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg127-server
unloading modules via unload_modules_local on: 'oleg127-server'
oleg127-server: oleg127-server.virtnet: executing unload_modules_local
oleg127-server: kmemleak disabled
modules unloaded.
=== sanity-scrub: start setup 07:10:17 (1788520217) ===
Stopping clients: oleg127-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg127-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 2
oleg127-server: oleg127-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1921056 Sep  3 15:04 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg127-server'
oleg127-server: oleg127-server.virtnet: executing load_modules_local
oleg127-server: kmemleak disabled
oleg127-server: Loading modules from /home/green/git/lustre-release/lustre
oleg127-server: detected 4 online CPUs by sysfs
oleg127-server: MODOPTS_LIBCFS=
oleg127-server: Force libcfs to create 2 CPU partitions
oleg127-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg127-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg127-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg127-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/vdc
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
Format mds2: /dev/vdd
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
Format ost1: /dev/vde
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
Format ost2: /dev/vdf
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
Checking servers environments
Checking clients oleg127-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg127-server'
oleg127-server: oleg127-server.virtnet: executing load_modules_local
oleg127-server: kmemleak disabled
oleg127-server: Loading modules from /home/green/git/lustre-release/lustre
oleg127-server: detected 4 online CPUs by sysfs
oleg127-server: MODOPTS_LIBCFS=
oleg127-server: Force libcfs to create 2 CPU partitions
oleg127-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Setup mgs, mdt, osts
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
Commit the device label on /dev/vdc
Started lustre-MDT0000
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
Commit the device label on /dev/vdd
Started lustre-MDT0001
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
Commit the device label on /dev/vde
Started lustre-OST0000
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
Commit the device label on /dev/vdf
Started lustre-OST0001
Starting client: oleg127-client.virtnet:  -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre
Starting client oleg127-client.virtnet:  -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre
Started clients oleg127-client.virtnet: 
192.168.201.127@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff94c012332800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff94c012332800.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
Updated after 3s: want 'procname_uid' got 'procname_uid'
disable quota as required
osd-ldiskfs.track_declares_assert=1
=== sanity-scrub: finish setup 07:13:50 (1788520430) ===

== sanity-scrub test 0: Do not auto trigger OI scrub for non-backup/restore case ========================================================== 07:13:53 (1788520433)
preparing... Fri Sep  4 07:13:56 EDT 2026
creating 0 files on mds1
striped dir -i0 -c1 -H all_char  /mnt/lustre/d0.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455478 s, 23.0 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H crush  /mnt/lustre/d0.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553475 s, 18.9 MB/s
prepared Fri Sep  4 07:14:09 EDT 2026.
stop mds1
stop mds2
starting MDTs without disabling OI scrub
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
Starting client: oleg127-client.virtnet:  -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre
PASS 0 (95s)

== sanity-scrub test 1a: Auto trigger initial OI scrub when server mounts ========================================================== 07:15:29 (1788520529)
preparing... Fri Sep  4 07:15:33 EDT 2026
creating 0 files on mds1
striped dir -i0 -c1 -H crush2  /mnt/lustre/d1a.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440751 s, 23.8 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H all_char  /mnt/lustre/d1a.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301953 s, 34.7 MB/s
prepared Fri Sep  4 07:15:42 EDT 2026.
stop mds1
stop mds2
start mds1 without disabling OI scrub: -o localrecov -o user_xattr
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
Starting client: oleg127-client.virtnet:  -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre
fail_loc=0x193
fail_loc=0
192.168.201.127@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg127-client.virtnet /mnt/lustre (opts:)
stop mds1
start mds1 with disabling OI scrub: -o localrecov -o user_xattr,noscrub
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
PASS 1a (87s)

== sanity-scrub test 1b: Trigger OI scrub when MDT mounts for OI files remove/recreate case ========================================================== 07:16:56 (1788520616)
Checking servers environments
Checking clients oleg127-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg127-server'
oleg127-server: oleg127-server.virtnet: executing load_modules_local
oleg127-server: kmemleak disabled
oleg127-server: Loading modules from /home/green/git/lustre-release/lustre
oleg127-server: detected 4 online CPUs by sysfs
oleg127-server: MODOPTS_LIBCFS=
oleg127-server: Force libcfs to create 2 CPU partitions
oleg127-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg127-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg127-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg127-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg127-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg127-client.virtnet:  -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre
Starting client oleg127-client.virtnet:  -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre
Started clients oleg127-client.virtnet: 
192.168.201.127@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff94c003808800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff94c003808800.idle_timeout=debug
disable quota as required
preparing... Fri Sep  4 07:18:01 EDT 2026
creating 0 files on mds1
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d1b.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079711 s, 13.2 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H all_char  /mnt/lustre/d1b.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526284 s, 19.9 MB/s
prepared Fri Sep  4 07:18:10 EDT 2026.
fail_loc=0x198
fail_loc=0
stop mds1
stop mds2
removing OI files on mds1: idx=
removed '/mnt/lustre-brpt/oi.16.0'
removed '/mnt/lustre-brpt/oi.16.1'
removed '/mnt/lustre-brpt/oi.16.10'
removed '/mnt/lustre-brpt/oi.16.11'
removed '/mnt/lustre-brpt/oi.16.12'
removed '/mnt/lustre-brpt/oi.16.13'
removed '/mnt/lustre-brpt/oi.16.14'
removed '/mnt/lustre-brpt/oi.16.15'
removed '/mnt/lustre-brpt/oi.16.16'
removed '/mnt/lustre-brpt/oi.16.17'
removed '/mnt/lustre-brpt/oi.16.18'
removed '/mnt/lustre-brpt/oi.16.19'
removed '/mnt/lustre-brpt/oi.16.2'
removed '/mnt/lustre-brpt/oi.16.20'
removed '/mnt/lustre-brpt/oi.16.21'
removed '/mnt/lustre-brpt/oi.16.22'
removed '/mnt/lustre-brpt/oi.16.23'
removed '/mnt/lustre-brpt/oi.16.24'
removed '/mnt/lustre-brpt/oi.16.25'
removed '/mnt/lustre-brpt/oi.16.26'
removed '/mnt/lustre-brpt/oi.16.27'
removed '/mnt/lustre-brpt/oi.16.28'
removed '/mnt/lustre-brpt/oi.16.29'
removed '/mnt/lustre-brpt/oi.16.3'
removed '/mnt/lustre-brpt/oi.16.30'
removed '/mnt/lustre-brpt/oi.16.31'
removed '/mnt/lustre-brpt/oi.16.32'
removed '/mnt/lustre-brpt/oi.16.33'
removed '/mnt/lustre-brpt/oi.16.34'
removed '/mnt/lustre-brpt/oi.16.35'
removed '/mnt/lustre-brpt/oi.16.36'
removed '/mnt/lustre-brpt/oi.16.37'
removed '/mnt/lustre-brpt/oi.16.38'
removed '/mnt/lustre-brpt/oi.16.39'
removed '/mnt/lustre-brpt/oi.16.4'
removed '/mnt/lustre-brpt/oi.16.40'
removed '/mnt/lustre-brpt/oi.16.41'
removed '/mnt/lustre-brpt/oi.16.42'
removed '/mnt/lustre-brpt/oi.16.43'
removed '/mnt/lustre-brpt/oi.16.44'
removed '/mnt/lustre-brpt/oi.16.45'
removed '/mnt/lustre-brpt/oi.16.46'
removed '/mnt/lustre-brpt/oi.16.47'
removed '/mnt/lustre-brpt/oi.16.48'
removed '/mnt/lustre-brpt/oi.16.49'
removed '/mnt/lustre-brpt/oi.16.5'
removed '/mnt/lustre-brpt/oi.16.50'
removed '/mnt/lustre-brpt/oi.16.51'
removed '/mnt/lustre-brpt/oi.16.52'
removed '/mnt/lustre-brpt/oi.16.53'
removed '/mnt/lustre-brpt/oi.16.54'
removed '/mnt/lustre-brpt/oi.16.55'
removed '/mnt/lustre-brpt/oi.16.56'
removed '/mnt/lustre-brpt/oi.16.57'
removed '/mnt/lustre-brpt/oi.16.58'
removed '/mnt/lustre-brpt/oi.16.59'
removed '/mnt/lustre-brpt/oi.16.6'
removed '/mnt/lustre-brpt/oi.16.60'
removed '/mnt/lustre-brpt/oi.16.61'
removed '/mnt/lustre-brpt/oi.16.62'
removed '/mnt/lustre-brpt/oi.16.63'
removed '/mnt/lustre-brpt/oi.16.7'
removed '/mnt/lustre-brpt/oi.16.8'
removed '/mnt/lustre-brpt/oi.16.9'
removing OI files on mds2: idx=
removed '/mnt/lustre-brpt/oi.16.0'
removed '/mnt/lustre-brpt/oi.16.1'
removed '/mnt/lustre-brpt/oi.16.10'
removed '/mnt/lustre-brpt/oi.16.11'
removed '/mnt/lustre-brpt/oi.16.12'
removed '/mnt/lustre-brpt/oi.16.13'
removed '/mnt/lustre-brpt/oi.16.14'
removed '/mnt/lustre-brpt/oi.16.15'
removed '/mnt/lustre-brpt/oi.16.16'
removed '/mnt/lustre-brpt/oi.16.17'
removed '/mnt/lustre-brpt/oi.16.18'
removed '/mnt/lustre-brpt/oi.16.19'
removed '/mnt/lustre-brpt/oi.16.2'
removed '/mnt/lustre-brpt/oi.16.20'
removed '/mnt/lustre-brpt/oi.16.21'
removed '/mnt/lustre-brpt/oi.16.22'
removed '/mnt/lustre-brpt/oi.16.23'
removed '/mnt/lustre-brpt/oi.16.24'
removed '/mnt/lustre-brpt/oi.16.25'
removed '/mnt/lustre-brpt/oi.16.26'
removed '/mnt/lustre-brpt/oi.16.27'
removed '/mnt/lustre-brpt/oi.16.28'
removed '/mnt/lustre-brpt/oi.16.29'
removed '/mnt/lustre-brpt/oi.16.3'
removed '/mnt/lustre-brpt/oi.16.30'
removed '/mnt/lustre-brpt/oi.16.31'
removed '/mnt/lustre-brpt/oi.16.32'
removed '/mnt/lustre-brpt/oi.16.33'
removed '/mnt/lustre-brpt/oi.16.34'
removed '/mnt/lustre-brpt/oi.16.35'
removed '/mnt/lustre-brpt/oi.16.36'
removed '/mnt/lustre-brpt/oi.16.37'
removed '/mnt/lustre-brpt/oi.16.38'
removed '/mnt/lustre-brpt/oi.16.39'
removed '/mnt/lustre-brpt/oi.16.4'
removed '/mnt/lustre-brpt/oi.16.40'
removed '/mnt/lustre-brpt/oi.16.41'
removed '/mnt/lustre-brpt/oi.16.42'
removed '/mnt/lustre-brpt/oi.16.43'
removed '/mnt/lustre-brpt/oi.16.44'
removed '/mnt/lustre-brpt/oi.16.45'
removed '/mnt/lustre-brpt/oi.16.46'
removed '/mnt/lustre-brpt/oi.16.47'
removed '/mnt/lustre-brpt/oi.16.48'
removed '/mnt/lustre-brpt/oi.16.49'
removed '/mnt/lustre-brpt/oi.16.5'
removed '/mnt/lustre-brpt/oi.16.50'
removed '/mnt/lustre-brpt/oi.16.51'
removed '/mnt/lustre-brpt/oi.16.52'
removed '/mnt/lustre-brpt/oi.16.53'
removed '/mnt/lustre-brpt/oi.16.54'
removed '/mnt/lustre-brpt/oi.16.55'
removed '/mnt/lustre-brpt/oi.16.56'
removed '/mnt/lustre-brpt/oi.16.57'
removed '/mnt/lustre-brpt/oi.16.58'
removed '/mnt/lustre-brpt/oi.16.59'
removed '/mnt/lustre-brpt/oi.16.6'
removed '/mnt/lustre-brpt/oi.16.60'
removed '/mnt/lustre-brpt/oi.16.61'
removed '/mnt/lustre-brpt/oi.16.62'
removed '/mnt/lustre-brpt/oi.16.63'
removed '/mnt/lustre-brpt/oi.16.7'
removed '/mnt/lustre-brpt/oi.16.8'
removed '/mnt/lustre-brpt/oi.16.9'
start MDTs without disabling OI scrub
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
Starting client: oleg127-client.virtnet:  -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre
PASS 1b (151s)

== sanity-scrub test 1c: Auto detect kinds of OI file(s) removed/recreated cases ========================================================== 07:19:27 (1788520767)
preparing... Fri Sep  4 07:19:31 EDT 2026
creating 0 files on mds1
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d1c.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314121 s, 33.4 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H all_char  /mnt/lustre/d1c.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037912 s, 27.7 MB/s
prepared Fri Sep  4 07:19:40 EDT 2026.
stop mds1
stop mds2
removing OI files on mds1: idx=0
removed '/mnt/lustre-brpt/oi.16.0'
removing OI files on mds2: idx=0
removed '/mnt/lustre-brpt/oi.16.0'
start MDTs with OI scrub disabled
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
Started LFSCK on the device lustre-MDT0000: scrub
Started LFSCK on the device lustre-MDT0001: scrub
Checking servers environments
Checking clients oleg127-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg127-server'
oleg127-server: oleg127-server.virtnet: executing load_modules_local
oleg127-server: kmemleak disabled
oleg127-server: Loading modules from /home/green/git/lustre-release/lustre
oleg127-server: detected 4 online CPUs by sysfs
oleg127-server: MODOPTS_LIBCFS=
oleg127-server: Force libcfs to create 2 CPU partitions
oleg127-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg127-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg127-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg127-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg127-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg127-client.virtnet:  -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre
Starting client oleg127-client.virtnet:  -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre
Started clients oleg127-client.virtnet: 
192.168.201.127@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff94c0099a9000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff94c0099a9000.idle_timeout=debug
disable quota as required
preparing... Fri Sep  4 07:21:47 EDT 2026
creating 0 files on mds1
striped dir -i0 -c1 -H crush2  /mnt/lustre/d1c.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317745 s, 33.0 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H crush  /mnt/lustre/d1c.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280547 s, 37.4 MB/s
prepared Fri Sep  4 07:21:55 EDT 2026.
stop mds1
stop mds2
removing OI files on mds1: idx=2
removed '/mnt/lustre-brpt/oi.16.2'
removed '/mnt/lustre-brpt/oi.16.4'
removed '/mnt/lustre-brpt/oi.16.8'
removed '/mnt/lustre-brpt/oi.16.16'
removed '/mnt/lustre-brpt/oi.16.32'
removing OI files on mds2: idx=2
removed '/mnt/lustre-brpt/oi.16.2'
removed '/mnt/lustre-brpt/oi.16.4'
removed '/mnt/lustre-brpt/oi.16.8'
removed '/mnt/lustre-brpt/oi.16.16'
removed '/mnt/lustre-brpt/oi.16.32'
start MDTs with OI scrub disabled
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
Started LFSCK on the device lustre-MDT0000: scrub
Started LFSCK on the device lustre-MDT0001: scrub
Checking servers environments
Checking clients oleg127-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg127-server'
oleg127-server: oleg127-server.virtnet: executing load_modules_local
oleg127-server: kmemleak disabled
oleg127-server: Loading modules from /home/green/git/lustre-release/lustre
oleg127-server: detected 4 online CPUs by sysfs
oleg127-server: MODOPTS_LIBCFS=
oleg127-server: Force libcfs to create 2 CPU partitions
oleg127-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg127-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg127-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg127-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg127-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg127-client.virtnet:  -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre
Starting client oleg127-client.virtnet:  -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre
Started clients oleg127-client.virtnet: 
192.168.201.127@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff94c0099d7800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff94c0099d7800.idle_timeout=debug
disable quota as required
preparing... Fri Sep  4 07:23:57 EDT 2026
creating 0 files on mds1
striped dir -i0 -c1 -H all_char  /mnt/lustre/d1c.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030048 s, 34.9 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H crush2  /mnt/lustre/d1c.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400875 s, 26.2 MB/s
prepared Fri Sep  4 07:24:04 EDT 2026.
stop mds1
stop mds2
removing OI files on mds1: idx=3
removed '/mnt/lustre-brpt/oi.16.3'
removed '/mnt/lustre-brpt/oi.16.9'
removed '/mnt/lustre-brpt/oi.16.27'
removing OI files on mds2: idx=3
removed '/mnt/lustre-brpt/oi.16.3'
removed '/mnt/lustre-brpt/oi.16.9'
removed '/mnt/lustre-brpt/oi.16.27'
start MDTs with OI scrub disabled
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
Started LFSCK on the device lustre-MDT0000: scrub
Started LFSCK on the device lustre-MDT0001: scrub
PASS 1c (358s)

== sanity-scrub test 2: Trigger OI scrub when MDT mounts for backup/restore case ========================================================== 07:25:25 (1788521125)
Checking servers environments
Checking clients oleg127-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg127-server'
oleg127-server: oleg127-server.virtnet: executing load_modules_local
oleg127-server: kmemleak disabled
oleg127-server: Loading modules from /home/green/git/lustre-release/lustre
oleg127-server: detected 4 online CPUs by sysfs
oleg127-server: MODOPTS_LIBCFS=
oleg127-server: Force libcfs to create 2 CPU partitions
oleg127-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg127-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg127-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg127-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg127-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg127-client.virtnet:  -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre
Starting client oleg127-client.virtnet:  -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre
Started clients oleg127-client.virtnet: 
192.168.201.127@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff94c012334000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff94c012334000.idle_timeout=debug
disable quota as required
preparing... Fri Sep  4 07:26:28 EDT 2026
creating 0 files on mds1
striped dir -i0 -c1 -H crush2  /mnt/lustre/d2.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327043 s, 32.1 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H fnv_1a_64  /mnt/lustre/d2.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021716 s, 48.3 MB/s
prepared Fri Sep  4 07:26:37 EDT 2026.
stop mds1
stop mds2
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
starting MDTs without disabling OI scrub
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
Starting client: oleg127-client.virtnet:  -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre
PASS 2 (196s)

== sanity-scrub test 4a: Auto trigger OI scrub if bad OI mapping was found (1) ========================================================== 07:28:41 (1788521321)
preparing... Fri Sep  4 07:28:46 EDT 2026
creating 0 files on mds1
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d4a.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478495 s, 21.9 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H all_char  /mnt/lustre/d4a.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330148 s, 31.8 MB/s
prepared Fri Sep  4 07:28:54 EDT 2026.
stop mds1
stop mds2
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
starting MDTs with OI scrub disabled
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
Starting client: oleg127-client.virtnet:  -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre
PASS 4a (119s)

== sanity-scrub test 4b: Auto trigger OI scrub if bad OI mapping was found (2) ========================================================== 07:30:41 (1788521441)
preparing... Fri Sep  4 07:30:45 EDT 2026
creating 5 files on mds1
striped dir -i0 -c1 -H all_char  /mnt/lustre/d4b.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336512 s, 31.2 MB/s
creating 5 files on mds2
striped dir -i1 -c1 -H fnv_1a_64  /mnt/lustre/d4b.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041969 s, 25.0 MB/s
prepared Fri Sep  4 07:30:54 EDT 2026.
stop mds1
stop mds2
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
starting MDTs with OI scrub disabled
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
Starting client: oleg127-client.virtnet:  -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre
OI scrub on MDS1 status for the 1st time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 137 seconds
time_since_latest_start: 13 seconds
time_since_last_checkpoint: 13 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 3
updated: 0
failed: 0
prior_updated: 3
noscrub: 0
igif: 0
success_count: 6
run_time: 13 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS2 status for the 1st time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 137 seconds
time_since_latest_start: 11 seconds
time_since_last_checkpoint: 11 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 1
updated: 0
failed: 0
prior_updated: 1
noscrub: 0
igif: 0
success_count: 6
run_time: 11 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS1 status for the 2nd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 149 seconds
time_since_latest_start: 25 seconds
time_since_last_checkpoint: 25 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 4
updated: 0
failed: 0
prior_updated: 4
noscrub: 0
igif: 0
success_count: 6
run_time: 25 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS2 status for the 2nd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 150 seconds
time_since_latest_start: 24 seconds
time_since_last_checkpoint: 24 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 2
updated: 0
failed: 0
prior_updated: 2
noscrub: 0
igif: 0
success_count: 6
run_time: 24 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS1 status for the 3rd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 48 seconds
time_since_latest_start: 48 seconds
time_since_last_checkpoint: 48 seconds
latest_start_position: 12
last_checkpoint_position: 160001
first_failure_position: N/A
checked: 255
updated: 50
failed: 0
prior_updated: 38
noscrub: 70
igif: 1
success_count: 7
run_time: 0 seconds
average_speed: 255 objects/sec
real_time_speed: N/A
current_position: N/A
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS2 status for the 3rd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 14 seconds
time_since_latest_start: 14 seconds
time_since_last_checkpoint: 14 seconds
latest_start_position: 12
last_checkpoint_position: 160001
first_failure_position: N/A
checked: 234
updated: 52
failed: 0
prior_updated: 35
noscrub: 73
igif: 1
success_count: 7
run_time: 0 seconds
average_speed: 234 objects/sec
real_time_speed: N/A
current_position: N/A
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
total 1024
-rw-r--r-- 1 root root       0 Sep  4 07:30 f1
-rw-r--r-- 1 root root 1048576 Sep  4 07:30 f2
total 1024
-rw-r--r-- 1 root root       0 Sep  4 07:30 f1
-rw-r--r-- 1 root root 1048576 Sep  4 07:30 f2
PASS 4b (221s)

== sanity-scrub test 4c: Auto trigger OI scrub if bad OI mapping was found (3) ========================================================== 07:34:22 (1788521662)
preparing... Fri Sep  4 07:34:25 EDT 2026
creating 500 files on mds1
striped dir -i0 -c1 -H all_char  /mnt/lustre/d4c.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359163 s, 29.2 MB/s
creating 500 files on mds2
striped dir -i1 -c1 -H all_char  /mnt/lustre/d4c.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395096 s, 26.5 MB/s
prepared Fri Sep  4 07:34:44 EDT 2026.
stop mds1
stop mds2
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
starting MDTs with OI scrub disabled
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
Starting client: oleg127-client.virtnet:  -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre
OI scrub on MDS1 status for the 1st time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 206 seconds
time_since_latest_start: 14 seconds
time_since_last_checkpoint: 14 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 3
updated: 0
failed: 0
prior_updated: 3
noscrub: 0
igif: 0
success_count: 7
run_time: 14 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS2 status for the 1st time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 171 seconds
time_since_latest_start: 12 seconds
time_since_last_checkpoint: 12 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 1
updated: 0
failed: 0
prior_updated: 1
noscrub: 0
igif: 0
success_count: 7
run_time: 12 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS1 status for the 2nd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 219 seconds
time_since_latest_start: 27 seconds
time_since_last_checkpoint: 27 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 4
updated: 0
failed: 0
prior_updated: 4
noscrub: 0
igif: 0
success_count: 7
run_time: 27 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS2 status for the 2nd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 186 seconds
time_since_latest_start: 27 seconds
time_since_last_checkpoint: 27 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 2
updated: 0
failed: 0
prior_updated: 2
noscrub: 0
igif: 0
success_count: 7
run_time: 27 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS1 status for the 3rd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 34 seconds
time_since_latest_start: 34 seconds
time_since_last_checkpoint: 34 seconds
latest_start_position: 12
last_checkpoint_position: 160001
first_failure_position: N/A
checked: 752
updated: 562
failed: 0
prior_updated: 21
noscrub: 70
igif: 1
success_count: 8
run_time: 0 seconds
average_speed: 752 objects/sec
real_time_speed: N/A
current_position: N/A
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS2 status for the 3rd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 14 seconds
time_since_latest_start: 14 seconds
time_since_last_checkpoint: 14 seconds
latest_start_position: 12
last_checkpoint_position: 160001
first_failure_position: N/A
checked: 731
updated: 561
failed: 0
prior_updated: 21
noscrub: 73
igif: 1
success_count: 8
run_time: 0 seconds
average_speed: 731 objects/sec
real_time_speed: N/A
current_position: N/A
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
-r--r--r-- 1 root root 0 Sep  4 07:34 /mnt/lustre/d4c.sanity-scrub/mds1/f4c.sanity-scrub1
-r--r--r-- 1 root root 0 Sep  4 07:34 /mnt/lustre/d4c.sanity-scrub/mds2/f4c.sanity-scrub1
PASS 4c (226s)

== sanity-scrub test 4d: FID in LMA mismatch with object FID won't block create ========================================================== 07:38:09 (1788521889)
fail_loc=0x19b
fail_loc=0
PASS 4d (35s)

== sanity-scrub test 4e: FID reuse can be fixed ========== 07:38:44 (1788521924)
 - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 801 prealloc_next_id: 774 count: 28
fail_loc=0x1a0
fail_loc=0
Started LFSCK on the device lustre-OST0000: scrub
PASS 4e (16s)

== sanity-scrub test 5: OI scrub state machine =========== 07:39:00 (1788521940)
oleg127-server: oleg127-server.virtnet: executing set_hostid
oleg127-server: oleg127-server.virtnet: executing load_modules_local
oleg127-server: oleg127-server.virtnet: executing load_modules_local
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
Using TIMEOUT=20
preparing... Fri Sep  4 07:42:37 EDT 2026
creating 100 files on mds1
striped dir -i0 -c1 -H all_char  /mnt/lustre/d5.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269257 s, 38.9 MB/s
creating 100 files on mds2
striped dir -i1 -c1 -H fnv_1a_64  /mnt/lustre/d5.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673194 s, 15.6 MB/s
prepared Fri Sep  4 07:42:49 EDT 2026.
stop mds1
stop mds2
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
starting MDTs with OI scrub disabled (1)
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1
oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1