-----============= acceptance-small: sanity-scrub ============----- Tue Jun 23 09:47:32 EDT 2026
MGS_OS_ID_LIKE=rhel centos fedora rocky
MGS_OS_VERSION_ID=8.10
MGS_OS_ID=rocky
MGS_OS_VERSION_CODE=134873088
MDS1_OS_VERSION_ID=8.10
MDS1_OS_VERSION_CODE=134873088
MDS1_OS_ID_LIKE=rhel centos fedora rocky
MDS1_OS_ID=rocky
OST1_OS_VERSION_CODE=134873088
OST1_OS_ID_LIKE=rhel centos fedora rocky
OST1_OS_VERSION_ID=8.10
OST1_OS_ID=rocky
CLIENT_OS_ID=rocky
CLIENT_OS_VERSION_CODE=134873088
CLIENT_OS_VERSION_ID=8.10
CLIENT_OS_ID_LIKE=rhel centos fedora rocky
oleg132-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-scrub.*ex': No such file or directory
excepting tests: 
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg132-server'
oleg132-server: oleg132-server.virtnet: executing load_modules_local
oleg132-server: Loading modules from /home/green/git/lustre-release/lustre
oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg132-server: detected 4 online CPUs by sysfs
oleg132-server: MODOPTS_LIBCFS=
oleg132-server: Force libcfs to create 2 CPU partitions
client=34682451 MDS=34682451 OSS=34682451
Stopping clients: oleg132-client.virtnet /mnt/lustre (opts:)
Stopping client oleg132-client.virtnet /mnt/lustre opts:
Stopping clients: oleg132-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg132-server

unloading modules via unload_modules_local on: 'oleg132-server'
oleg132-server: oleg132-server.virtnet: executing unload_modules_local
oleg132-server: 
modules unloaded.
=== sanity-scrub: start setup 09:50:02 (1782222602) ===
Stopping clients: oleg132-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg132-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2
oleg132-server: oleg132-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg132-server'
oleg132-server: oleg132-server.virtnet: executing load_modules_local
oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg132-server: Loading modules from /home/green/git/lustre-release/lustre
oleg132-server: detected 4 online CPUs by sysfs
oleg132-server: MODOPTS_LIBCFS=
oleg132-server: Force libcfs to create 2 CPU partitions
oleg132-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg132-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg132-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Formatting mgs, mds, osts
Format mds1: /dev/vdc
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Format mds2: /dev/vdd
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Format ost1: /dev/vde
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Format ost2: /dev/vdf
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Checking servers environments
Checking clients oleg132-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg132-server'
oleg132-server: oleg132-server.virtnet: executing load_modules_local
oleg132-server: Loading modules from /home/green/git/lustre-release/lustre
oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg132-server: detected 4 online CPUs by sysfs
oleg132-server: MODOPTS_LIBCFS=
oleg132-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Commit the device label on /dev/vdc
Started lustre-MDT0000
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Commit the device label on /dev/vdd
Started lustre-MDT0001
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Commit the device label on /dev/vde
Started lustre-OST0000
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Commit the device label on /dev/vdf
Started lustre-OST0001
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
Starting client oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
Started clients oleg132-client.virtnet: 
192.168.201.132@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8e328909e800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8e328909e800.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
disable quota as required
osd-ldiskfs.track_declares_assert=1
=== sanity-scrub: finish setup 09:54:33 (1782222873) ===

== sanity-scrub test 0: Do not auto trigger OI scrub for non-backup/restore case ========================================================== 09:54:35 (1782222875)
preparing... Tue Jun 23 09:54:38 EDT 2026
creating 0 files on mds1
striped dir -i0 -c1 -H crush2  /mnt/lustre/d0.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596503 s, 17.6 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H crush  /mnt/lustre/d0.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441933 s, 23.7 MB/s
prepared Tue Jun 23 09:54:52 EDT 2026.
stop mds1
stop mds2
starting MDTs without disabling OI scrub
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
PASS 0 (87s)

== sanity-scrub test 1a: Auto trigger initial OI scrub when server mounts ========================================================== 09:56:02 (1782222962)
preparing... Tue Jun 23 09:56:08 EDT 2026
creating 0 files on mds1
striped dir -i0 -c1 -H all_char  /mnt/lustre/d1a.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354568 s, 29.6 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H all_char  /mnt/lustre/d1a.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054806 s, 19.1 MB/s
prepared Tue Jun 23 09:56:21 EDT 2026.
stop mds1
stop mds2
start mds1 without disabling OI scrub: -o localrecov -o user_xattr
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
fail_loc=0x193
fail_loc=0
192.168.201.132@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg132-client.virtnet /mnt/lustre (opts:)
stop mds1
start mds1 with disabling OI scrub: -o localrecov -o user_xattr,noscrub
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
PASS 1a (131s)

== sanity-scrub test 1b: Trigger OI scrub when MDT mounts for OI files remove/recreate case ========================================================== 09:58:13 (1782223093)
Checking servers environments
Checking clients oleg132-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg132-server'
oleg132-server: oleg132-server.virtnet: executing load_modules_local
oleg132-server: Loading modules from /home/green/git/lustre-release/lustre
oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg132-server: detected 4 online CPUs by sysfs
oleg132-server: MODOPTS_LIBCFS=
oleg132-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
Starting client oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
Started clients oleg132-client.virtnet: 
192.168.201.132@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8e32998ca000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8e32998ca000.idle_timeout=debug
disable quota as required
preparing... Tue Jun 23 09:59:34 EDT 2026
creating 0 files on mds1
striped dir -i0 -c1 -H crush  /mnt/lustre/d1b.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884088 s, 11.9 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H crush2  /mnt/lustre/d1b.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131235 s, 8.0 MB/s
prepared Tue Jun 23 09:59:47 EDT 2026.
fail_loc=0x198
fail_loc=0
stop mds1
stop mds2
removing OI files on mds1: idx=
removed '/mnt/lustre-brpt/oi.16.0'
removed '/mnt/lustre-brpt/oi.16.1'
removed '/mnt/lustre-brpt/oi.16.10'
removed '/mnt/lustre-brpt/oi.16.11'
removed '/mnt/lustre-brpt/oi.16.12'
removed '/mnt/lustre-brpt/oi.16.13'
removed '/mnt/lustre-brpt/oi.16.14'
removed '/mnt/lustre-brpt/oi.16.15'
removed '/mnt/lustre-brpt/oi.16.16'
removed '/mnt/lustre-brpt/oi.16.17'
removed '/mnt/lustre-brpt/oi.16.18'
removed '/mnt/lustre-brpt/oi.16.19'
removed '/mnt/lustre-brpt/oi.16.2'
removed '/mnt/lustre-brpt/oi.16.20'
removed '/mnt/lustre-brpt/oi.16.21'
removed '/mnt/lustre-brpt/oi.16.22'
removed '/mnt/lustre-brpt/oi.16.23'
removed '/mnt/lustre-brpt/oi.16.24'
removed '/mnt/lustre-brpt/oi.16.25'
removed '/mnt/lustre-brpt/oi.16.26'
removed '/mnt/lustre-brpt/oi.16.27'
removed '/mnt/lustre-brpt/oi.16.28'
removed '/mnt/lustre-brpt/oi.16.29'
removed '/mnt/lustre-brpt/oi.16.3'
removed '/mnt/lustre-brpt/oi.16.30'
removed '/mnt/lustre-brpt/oi.16.31'
removed '/mnt/lustre-brpt/oi.16.32'
removed '/mnt/lustre-brpt/oi.16.33'
removed '/mnt/lustre-brpt/oi.16.34'
removed '/mnt/lustre-brpt/oi.16.35'
removed '/mnt/lustre-brpt/oi.16.36'
removed '/mnt/lustre-brpt/oi.16.37'
removed '/mnt/lustre-brpt/oi.16.38'
removed '/mnt/lustre-brpt/oi.16.39'
removed '/mnt/lustre-brpt/oi.16.4'
removed '/mnt/lustre-brpt/oi.16.40'
removed '/mnt/lustre-brpt/oi.16.41'
removed '/mnt/lustre-brpt/oi.16.42'
removed '/mnt/lustre-brpt/oi.16.43'
removed '/mnt/lustre-brpt/oi.16.44'
removed '/mnt/lustre-brpt/oi.16.45'
removed '/mnt/lustre-brpt/oi.16.46'
removed '/mnt/lustre-brpt/oi.16.47'
removed '/mnt/lustre-brpt/oi.16.48'
removed '/mnt/lustre-brpt/oi.16.49'
removed '/mnt/lustre-brpt/oi.16.5'
removed '/mnt/lustre-brpt/oi.16.50'
removed '/mnt/lustre-brpt/oi.16.51'
removed '/mnt/lustre-brpt/oi.16.52'
removed '/mnt/lustre-brpt/oi.16.53'
removed '/mnt/lustre-brpt/oi.16.54'
removed '/mnt/lustre-brpt/oi.16.55'
removed '/mnt/lustre-brpt/oi.16.56'
removed '/mnt/lustre-brpt/oi.16.57'
removed '/mnt/lustre-brpt/oi.16.58'
removed '/mnt/lustre-brpt/oi.16.59'
removed '/mnt/lustre-brpt/oi.16.6'
removed '/mnt/lustre-brpt/oi.16.60'
removed '/mnt/lustre-brpt/oi.16.61'
removed '/mnt/lustre-brpt/oi.16.62'
removed '/mnt/lustre-brpt/oi.16.63'
removed '/mnt/lustre-brpt/oi.16.7'
removed '/mnt/lustre-brpt/oi.16.8'
removed '/mnt/lustre-brpt/oi.16.9'
removing OI files on mds2: idx=
removed '/mnt/lustre-brpt/oi.16.0'
removed '/mnt/lustre-brpt/oi.16.1'
removed '/mnt/lustre-brpt/oi.16.10'
removed '/mnt/lustre-brpt/oi.16.11'
removed '/mnt/lustre-brpt/oi.16.12'
removed '/mnt/lustre-brpt/oi.16.13'
removed '/mnt/lustre-brpt/oi.16.14'
removed '/mnt/lustre-brpt/oi.16.15'
removed '/mnt/lustre-brpt/oi.16.16'
removed '/mnt/lustre-brpt/oi.16.17'
removed '/mnt/lustre-brpt/oi.16.18'
removed '/mnt/lustre-brpt/oi.16.19'
removed '/mnt/lustre-brpt/oi.16.2'
removed '/mnt/lustre-brpt/oi.16.20'
removed '/mnt/lustre-brpt/oi.16.21'
removed '/mnt/lustre-brpt/oi.16.22'
removed '/mnt/lustre-brpt/oi.16.23'
removed '/mnt/lustre-brpt/oi.16.24'
removed '/mnt/lustre-brpt/oi.16.25'
removed '/mnt/lustre-brpt/oi.16.26'
removed '/mnt/lustre-brpt/oi.16.27'
removed '/mnt/lustre-brpt/oi.16.28'
removed '/mnt/lustre-brpt/oi.16.29'
removed '/mnt/lustre-brpt/oi.16.3'
removed '/mnt/lustre-brpt/oi.16.30'
removed '/mnt/lustre-brpt/oi.16.31'
removed '/mnt/lustre-brpt/oi.16.32'
removed '/mnt/lustre-brpt/oi.16.33'
removed '/mnt/lustre-brpt/oi.16.34'
removed '/mnt/lustre-brpt/oi.16.35'
removed '/mnt/lustre-brpt/oi.16.36'
removed '/mnt/lustre-brpt/oi.16.37'
removed '/mnt/lustre-brpt/oi.16.38'
removed '/mnt/lustre-brpt/oi.16.39'
removed '/mnt/lustre-brpt/oi.16.4'
removed '/mnt/lustre-brpt/oi.16.40'
removed '/mnt/lustre-brpt/oi.16.41'
removed '/mnt/lustre-brpt/oi.16.42'
removed '/mnt/lustre-brpt/oi.16.43'
removed '/mnt/lustre-brpt/oi.16.44'
removed '/mnt/lustre-brpt/oi.16.45'
removed '/mnt/lustre-brpt/oi.16.46'
removed '/mnt/lustre-brpt/oi.16.47'
removed '/mnt/lustre-brpt/oi.16.48'
removed '/mnt/lustre-brpt/oi.16.49'
removed '/mnt/lustre-brpt/oi.16.5'
removed '/mnt/lustre-brpt/oi.16.50'
removed '/mnt/lustre-brpt/oi.16.51'
removed '/mnt/lustre-brpt/oi.16.52'
removed '/mnt/lustre-brpt/oi.16.53'
removed '/mnt/lustre-brpt/oi.16.54'
removed '/mnt/lustre-brpt/oi.16.55'
removed '/mnt/lustre-brpt/oi.16.56'
removed '/mnt/lustre-brpt/oi.16.57'
removed '/mnt/lustre-brpt/oi.16.58'
removed '/mnt/lustre-brpt/oi.16.59'
removed '/mnt/lustre-brpt/oi.16.6'
removed '/mnt/lustre-brpt/oi.16.60'
removed '/mnt/lustre-brpt/oi.16.61'
removed '/mnt/lustre-brpt/oi.16.62'
removed '/mnt/lustre-brpt/oi.16.63'
removed '/mnt/lustre-brpt/oi.16.7'
removed '/mnt/lustre-brpt/oi.16.8'
removed '/mnt/lustre-brpt/oi.16.9'
start MDTs without disabling OI scrub
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
PASS 1b (185s)

== sanity-scrub test 1c: Auto detect kinds of OI file(s) removed/recreated cases ========================================================== 10:01:19 (1782223279)
preparing... Tue Jun 23 10:01:26 EDT 2026
creating 0 files on mds1
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d1c.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264953 s, 39.6 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H all_char  /mnt/lustre/d1c.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425961 s, 24.6 MB/s
prepared Tue Jun 23 10:01:40 EDT 2026.
stop mds1
stop mds2
removing OI files on mds1: idx=0
removed '/mnt/lustre-brpt/oi.16.0'
removing OI files on mds2: idx=0
removed '/mnt/lustre-brpt/oi.16.0'
start MDTs with OI scrub disabled
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Started LFSCK on the device lustre-MDT0000: scrub
Started LFSCK on the device lustre-MDT0001: scrub
Checking servers environments
Checking clients oleg132-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg132-server'
oleg132-server: oleg132-server.virtnet: executing load_modules_local
oleg132-server: Loading modules from /home/green/git/lustre-release/lustre
oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg132-server: detected 4 online CPUs by sysfs
oleg132-server: MODOPTS_LIBCFS=
oleg132-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
Starting client oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
Started clients oleg132-client.virtnet: 
192.168.201.132@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8e3288e96000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8e3288e96000.idle_timeout=debug
disable quota as required
preparing... Tue Jun 23 10:04:47 EDT 2026
creating 0 files on mds1
striped dir -i0 -c1 -H crush2  /mnt/lustre/d1c.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471902 s, 22.2 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H crush  /mnt/lustre/d1c.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103516 s, 10.1 MB/s
prepared Tue Jun 23 10:05:02 EDT 2026.
stop mds1
stop mds2
removing OI files on mds1: idx=2
removed '/mnt/lustre-brpt/oi.16.2'
removed '/mnt/lustre-brpt/oi.16.4'
removed '/mnt/lustre-brpt/oi.16.8'
removed '/mnt/lustre-brpt/oi.16.16'
removed '/mnt/lustre-brpt/oi.16.32'
removing OI files on mds2: idx=2
removed '/mnt/lustre-brpt/oi.16.2'
removed '/mnt/lustre-brpt/oi.16.4'
removed '/mnt/lustre-brpt/oi.16.8'
removed '/mnt/lustre-brpt/oi.16.16'
removed '/mnt/lustre-brpt/oi.16.32'
start MDTs with OI scrub disabled
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Started LFSCK on the device lustre-MDT0000: scrub
Started LFSCK on the device lustre-MDT0001: scrub
Checking servers environments
Checking clients oleg132-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg132-server'
oleg132-server: oleg132-server.virtnet: executing load_modules_local
oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg132-server: Loading modules from /home/green/git/lustre-release/lustre
oleg132-server: detected 4 online CPUs by sysfs
oleg132-server: MODOPTS_LIBCFS=
oleg132-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
Starting client oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
Started clients oleg132-client.virtnet: 
192.168.201.132@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8e3290de7000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8e3290de7000.idle_timeout=debug
disable quota as required
preparing... Tue Jun 23 10:08:23 EDT 2026
creating 0 files on mds1
striped dir -i0 -c1 -H all_char  /mnt/lustre/d1c.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433517 s, 24.2 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H all_char  /mnt/lustre/d1c.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426873 s, 24.6 MB/s
prepared Tue Jun 23 10:08:37 EDT 2026.
stop mds1
stop mds2
removing OI files on mds1: idx=3
removed '/mnt/lustre-brpt/oi.16.3'
removed '/mnt/lustre-brpt/oi.16.9'
removed '/mnt/lustre-brpt/oi.16.27'
removing OI files on mds2: idx=3
removed '/mnt/lustre-brpt/oi.16.3'
removed '/mnt/lustre-brpt/oi.16.9'
removed '/mnt/lustre-brpt/oi.16.27'
start MDTs with OI scrub disabled
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Started LFSCK on the device lustre-MDT0000: scrub
Started LFSCK on the device lustre-MDT0001: scrub
PASS 1c (549s)

== sanity-scrub test 2: Trigger OI scrub when MDT mounts for backup/restore case ========================================================== 10:10:28 (1782223828)
Checking servers environments
Checking clients oleg132-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg132-server'
oleg132-server: oleg132-server.virtnet: executing load_modules_local
oleg132-server: Loading modules from /home/green/git/lustre-release/lustre
oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg132-server: detected 4 online CPUs by sysfs
oleg132-server: MODOPTS_LIBCFS=
oleg132-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
Starting client oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
Started clients oleg132-client.virtnet: 
192.168.201.132@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8e328405d800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8e328405d800.idle_timeout=debug
disable quota as required
preparing... Tue Jun 23 10:12:11 EDT 2026
creating 0 files on mds1
striped dir -i0 -c1 -H crush  /mnt/lustre/d2.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471048 s, 22.3 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H fnv_1a_64  /mnt/lustre/d2.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747305 s, 14.0 MB/s
prepared Tue Jun 23 10:12:23 EDT 2026.
stop mds1
stop mds2
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
starting MDTs without disabling OI scrub
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
PASS 2 (245s)

== sanity-scrub test 4a: Auto trigger OI scrub if bad OI mapping was found (1) ========================================================== 10:14:33 (1782224073)
preparing... Tue Jun 23 10:14:38 EDT 2026
creating 0 files on mds1
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d4a.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477931 s, 21.9 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H all_char  /mnt/lustre/d4a.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067033 s, 15.6 MB/s
prepared Tue Jun 23 10:14:51 EDT 2026.
stop mds1
stop mds2
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
starting MDTs with OI scrub disabled
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
PASS 4a (176s)

== sanity-scrub test 4b: Auto trigger OI scrub if bad OI mapping was found (2) ========================================================== 10:17:30 (1782224250)
preparing... Tue Jun 23 10:17:36 EDT 2026
creating 5 files on mds1
striped dir -i0 -c1 -H crush  /mnt/lustre/d4b.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826126 s, 12.7 MB/s
creating 5 files on mds2
striped dir -i1 -c1 -H crush2  /mnt/lustre/d4b.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044529 s, 23.5 MB/s
prepared Tue Jun 23 10:17:48 EDT 2026.
stop mds1
stop mds2
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
starting MDTs with OI scrub disabled
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
OI scrub on MDS1 status for the 1st time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 184 seconds
time_since_latest_start: 14 seconds
time_since_last_checkpoint: 14 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 3
updated: 0
failed: 0
prior_updated: 3
noscrub: 0
igif: 0
success_count: 6
run_time: 14 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS2 status for the 1st time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 185 seconds
time_since_latest_start: 14 seconds
time_since_last_checkpoint: 14 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 1
updated: 0
failed: 0
prior_updated: 1
noscrub: 0
igif: 0
success_count: 6
run_time: 14 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS1 status for the 2nd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 199 seconds
time_since_latest_start: 29 seconds
time_since_last_checkpoint: 29 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 4
updated: 0
failed: 0
prior_updated: 4
noscrub: 0
igif: 0
success_count: 6
run_time: 29 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS2 status for the 2nd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 200 seconds
time_since_latest_start: 29 seconds
time_since_last_checkpoint: 29 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 2
updated: 0
failed: 0
prior_updated: 2
noscrub: 0
igif: 0
success_count: 6
run_time: 29 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS1 status for the 3rd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 50 seconds
time_since_latest_start: 50 seconds
time_since_last_checkpoint: 50 seconds
latest_start_position: 12
last_checkpoint_position: 160001
first_failure_position: N/A
checked: 255
updated: 50
failed: 0
prior_updated: 38
noscrub: 70
igif: 1
success_count: 7
run_time: 0 seconds
average_speed: 255 objects/sec
real_time_speed: N/A
current_position: N/A
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS2 status for the 3rd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 15 seconds
time_since_latest_start: 15 seconds
time_since_last_checkpoint: 15 seconds
latest_start_position: 12
last_checkpoint_position: 160001
first_failure_position: N/A
checked: 234
updated: 52
failed: 0
prior_updated: 35
noscrub: 73
igif: 1
success_count: 7
run_time: 0 seconds
average_speed: 234 objects/sec
real_time_speed: N/A
current_position: N/A
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
total 1024
-rw-r--r-- 1 root root       0 Jun 23 10:17 f1
-rw-r--r-- 1 root root 1048576 Jun 23 10:17 f2
total 1024
-rw-r--r-- 1 root root       0 Jun 23 10:17 f1
-rw-r--r-- 1 root root 1048576 Jun 23 10:17 f2
PASS 4b (274s)

== sanity-scrub test 4c: Auto trigger OI scrub if bad OI mapping was found (3) ========================================================== 10:22:03 (1782224523)
preparing... Tue Jun 23 10:22:09 EDT 2026
creating 500 files on mds1
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d4c.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776888 s, 13.5 MB/s
creating 500 files on mds2
striped dir -i1 -c1 -H fnv_1a_64  /mnt/lustre/d4c.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862562 s, 12.2 MB/s
prepared Tue Jun 23 10:22:42 EDT 2026.
stop mds1
stop mds2
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
starting MDTs with OI scrub disabled
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
OI scrub on MDS1 status for the 1st time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 270 seconds
time_since_latest_start: 15 seconds
time_since_last_checkpoint: 15 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 3
updated: 0
failed: 0
prior_updated: 3
noscrub: 0
igif: 0
success_count: 7
run_time: 15 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS2 status for the 1st time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 236 seconds
time_since_latest_start: 15 seconds
time_since_last_checkpoint: 15 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 1
updated: 0
failed: 0
prior_updated: 1
noscrub: 0
igif: 0
success_count: 7
run_time: 16 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS1 status for the 2nd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 289 seconds
time_since_latest_start: 34 seconds
time_since_last_checkpoint: 34 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 4
updated: 0
failed: 0
prior_updated: 4
noscrub: 0
igif: 0
success_count: 7
run_time: 34 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS2 status for the 2nd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 254 seconds
time_since_latest_start: 33 seconds
time_since_last_checkpoint: 33 seconds
latest_start_position: 12
last_checkpoint_position: 11
first_failure_position: N/A
checked: 2
updated: 0
failed: 0
prior_updated: 2
noscrub: 0
igif: 0
success_count: 7
run_time: 34 seconds
average_speed: 0 objects/sec
real_time_speed: 0 objects/sec
current_position: 12
scrub_in_prior: no
scrub_full_speed: no
partial_scan: yes
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS1 status for the 3rd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 37 seconds
time_since_latest_start: 37 seconds
time_since_last_checkpoint: 37 seconds
latest_start_position: 12
last_checkpoint_position: 160001
first_failure_position: N/A
checked: 752
updated: 562
failed: 0
prior_updated: 21
noscrub: 70
igif: 1
success_count: 8
run_time: 0 seconds
average_speed: 752 objects/sec
real_time_speed: N/A
current_position: N/A
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
OI scrub on MDS2 status for the 3rd time:
name: OI_scrub
magic: 0x4c5fe253
oi_files: 64
status: completed
flags:
param:
time_since_last_completed: 19 seconds
time_since_latest_start: 19 seconds
time_since_last_checkpoint: 19 seconds
latest_start_position: 12
last_checkpoint_position: 160001
first_failure_position: N/A
checked: 731
updated: 561
failed: 0
prior_updated: 21
noscrub: 73
igif: 1
success_count: 8
run_time: 0 seconds
average_speed: 731 objects/sec
real_time_speed: N/A
current_position: N/A
lf_scanned: 0
lf_repaired: 0
lf_failed: 0
-r--r--r-- 1 root root 0 Jun 23 10:22 /mnt/lustre/d4c.sanity-scrub/mds1/f4c.sanity-scrub1
-r--r--r-- 1 root root 0 Jun 23 10:22 /mnt/lustre/d4c.sanity-scrub/mds2/f4c.sanity-scrub1
PASS 4c (298s)

== sanity-scrub test 4d: FID in LMA mismatch with object FID won't block create ========================================================== 10:27:02 (1782224822)
fail_loc=0x19b
fail_loc=0
PASS 4d (52s)

== sanity-scrub test 4e: FID reuse can be fixed ========== 10:27:55 (1782224875)
 - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 801 prealloc_next_id: 774 count: 28
fail_loc=0x1a0
fail_loc=0
Started LFSCK on the device lustre-OST0000: scrub
PASS 4e (28s)

== sanity-scrub test 5: OI scrub state machine =========== 10:28:23 (1782224903)
oleg132-server: oleg132-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg132-server: oleg132-server.virtnet: executing load_modules_local
oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg132-server: oleg132-server.virtnet: executing load_modules_local
oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Using TIMEOUT=20
preparing... Tue Jun 23 10:32:43 EDT 2026
creating 100 files on mds1
striped dir -i0 -c1 -H all_char  /mnt/lustre/d5.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373193 s, 28.1 MB/s
creating 100 files on mds2
striped dir -i1 -c1 -H all_char  /mnt/lustre/d5.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100899 s, 10.4 MB/s
prepared Tue Jun 23 10:32:58 EDT 2026.
stop mds1
stop mds2
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
starting MDTs with OI scrub disabled (1)
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
fail_val=3
fail_loc=0x190
192.168.201.132@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg132-client.virtnet /mnt/lustre (opts:)
fail_loc=0x191
stopping mds1
stopping mds2
fail_loc=0
fail_val=0
starting MDTs with OI scrub disabled (2)
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
stopping mds1
stopping mds2
fail_val=3
fail_loc=0x190
starting MDTs without disabling OI scrub
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
fail_loc=0x192
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
fail_loc=0
fail_val=0
  File: /mnt/lustre/d5.sanity-scrub/mds1/sanity-scrub.sh
  Size: 42517     	Blocks: 88         IO Block: 4194304 regular file
Device: 2c54f966h/743766374d	Inode: 144115205289279557  Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2026-06-23 10:33:42.000000000 -0400
Modify: 2026-06-23 10:32:48.000000000 -0400
Change: 2026-06-23 10:33:42.000000000 -0400
 Birth: 2026-06-23 10:33:42.000000000 -0400
  File: /mnt/lustre/d5.sanity-scrub/mds2/sanity-scrub.sh
  Size: 42517     	Blocks: 88         IO Block: 4194304 regular file
Device: 2c54f966h/743766374d	Inode: 162129603798761540  Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2026-06-23 10:34:06.000000000 -0400
Modify: 2026-06-23 10:32:56.000000000 -0400
Change: 2026-06-23 10:34:06.000000000 -0400
 Birth: 2026-06-23 10:34:06.000000000 -0400
PASS 5 (574s)

== sanity-scrub test 6: OI scrub resumes from last checkpoint ========================================================== 10:37:57 (1782225477)
preparing... Tue Jun 23 10:38:07 EDT 2026
creating 100 files on mds1
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d6.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361452 s, 29.0 MB/s
creating 100 files on mds2
striped dir -i1 -c1 -H fnv_1a_64  /mnt/lustre/d6.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147076 s, 7.1 MB/s
prepared Tue Jun 23 10:38:24 EDT 2026.
stop mds1
stop mds2
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
starting MDTs with OI scrub disabled
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
fail_val=2
fail_loc=0x190
fail_loc=0x192
fail_val=3
fail_loc=0x190
  File: /mnt/lustre/d6.sanity-scrub/mds1/sanity-scrub.sh
  Size: 42517     	Blocks: 88         IO Block: 4194304 regular file
Device: 2c54f966h/743766374d	Inode: 144115305952575557  Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2026-06-23 10:39:06.000000000 -0400
Modify: 2026-06-23 10:38:14.000000000 -0400
Change: 2026-06-23 10:39:06.000000000 -0400
 Birth: 2026-06-23 10:39:06.000000000 -0400
  File: /mnt/lustre/d6.sanity-scrub/mds2/sanity-scrub.sh
  Size: 42517     	Blocks: 88         IO Block: 4194304 regular file
Device: 2c54f966h/743766374d	Inode: 162129704445280324  Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2026-06-23 10:39:31.000000000 -0400
Modify: 2026-06-23 10:38:22.000000000 -0400
Change: 2026-06-23 10:39:31.000000000 -0400
 Birth: 2026-06-23 10:39:31.000000000 -0400
192.168.201.132@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg132-client.virtnet /mnt/lustre (opts:)
fail_loc=0x191
stopping mds1
stopping mds2
fail_val=3
fail_loc=0x190
starting MDTs without disabling OI scrub
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
fail_loc=0
fail_val=0
Waiting 6s for 'completed'
Updated after 2s: want 'completed' got 'completed'
PASS 6 (273s)

== sanity-scrub test 7: System is available during OI scrub scanning ========================================================== 10:42:30 (1782225750)
Checking servers environments
Checking clients oleg132-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg132-server'
oleg132-server: oleg132-server.virtnet: executing load_modules_local
oleg132-server: Loading modules from /home/green/git/lustre-release/lustre
oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg132-server: detected 4 online CPUs by sysfs
oleg132-server: MODOPTS_LIBCFS=
oleg132-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
Starting client oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
Started clients oleg132-client.virtnet: 
192.168.201.132@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8e3290de6800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8e3290de6800.idle_timeout=debug
disable quota as required
preparing... Tue Jun 23 10:44:01 EDT 2026
creating 500 files on mds1
striped dir -i0 -c1 -H crush  /mnt/lustre/d7.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922508 s, 11.4 MB/s
creating 500 files on mds2
striped dir -i1 -c1 -H fnv_1a_64  /mnt/lustre/d7.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885038 s, 11.8 MB/s
prepared Tue Jun 23 10:44:34 EDT 2026.
stop mds1
stop mds2
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
starting MDTs with OI scrub disabled
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
fail_val=3
fail_loc=0x190
  File: /mnt/lustre/d7.sanity-scrub/mds1/f7.sanity-scrub300
  Size: 0         	Blocks: 0          IO Block: 4194304 regular empty file
Device: 2c54f966h/743766374d	Inode: 144115373061439871  Links: 1
Access: (0444/-r--r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2026-06-23 10:45:17.000000000 -0400
Modify: 2026-06-23 10:44:17.000000000 -0400
Change: 2026-06-23 10:45:17.000000000 -0400
 Birth: 2026-06-23 10:45:17.000000000 -0400
  File: /mnt/lustre/d7.sanity-scrub/mds2/f7.sanity-scrub300
  Size: 0         	Blocks: 0          IO Block: 4194304 regular empty file
Device: 2c54f966h/743766374d	Inode: 162129771554144638  Links: 1
Access: (0444/-r--r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2026-06-23 10:45:44.000000000 -0400
Modify: 2026-06-23 10:44:32.000000000 -0400
Change: 2026-06-23 10:45:44.000000000 -0400
 Birth: 2026-06-23 10:45:44.000000000 -0400
fail_loc=0
fail_val=0
PASS 7 (275s)

== sanity-scrub test 8: Control OI scrub manually ======== 10:47:05 (1782226025)
preparing... Tue Jun 23 10:47:25 EDT 2026
creating 128 files on mds1
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d8.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332068 s, 31.6 MB/s
creating 128 files on mds2
striped dir -i1 -c1 -H crush2  /mnt/lustre/d8.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443802 s, 23.6 MB/s
prepared Tue Jun 23 10:47:50 EDT 2026.
stop mds1
stop mds2
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
starting MDTs with OI scrub disabled
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
fail_val=1
fail_loc=0x190
Started LFSCK on the device lustre-MDT0000: scrub
Started LFSCK on the device lustre-MDT0001: scrub
Stopped LFSCK on the device lustre-MDT0000.
Stopped LFSCK on the device lustre-MDT0001.
Started LFSCK on the device lustre-MDT0000: scrub
Started LFSCK on the device lustre-MDT0001: scrub
fail_loc=0
fail_val=0
PASS 8 (204s)

== sanity-scrub test 9: OI scrub speed control =========== 10:50:29 (1782226229)
Checking servers environments
Checking clients oleg132-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg132-server'
oleg132-server: oleg132-server.virtnet: executing load_modules_local
oleg132-server: Loading modules from /home/green/git/lustre-release/lustre
oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg132-server: detected 4 online CPUs by sysfs
oleg132-server: MODOPTS_LIBCFS=
oleg132-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
Starting client oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
Started clients oleg132-client.virtnet: 
192.168.201.132@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8e328b1af800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8e328b1af800.idle_timeout=debug
disable quota as required
preparing... Tue Jun 23 10:52:03 EDT 2026
creating 6000 files on mds1
striped dir -i0 -c1 -H crush  /mnt/lustre/d9.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046956 s, 22.3 MB/s
creating 6000 files on mds2
striped dir -i1 -c1 -H crush2  /mnt/lustre/d9.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927988 s, 11.3 MB/s
prepared Tue Jun 23 10:54:21 EDT 2026.
stop mds1
stop mds2
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
starting MDTs with OI scrub disabled
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Started LFSCK on the device lustre-MDT0000: scrub
Started LFSCK on the device lustre-MDT0001: scrub
Started LFSCK on the device lustre-MDT0000: scrub
Started LFSCK on the device lustre-MDT0001: scrub
PASS 9 (449s)

== sanity-scrub test 10a: non-stopped OI scrub should auto restarts after MDS remount (1) ========================================================== 10:57:58 (1782226678)
Checking servers environments
Checking clients oleg132-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg132-server'
oleg132-server: oleg132-server.virtnet: executing load_modules_local
oleg132-server: Loading modules from /home/green/git/lustre-release/lustre
oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg132-server: detected 4 online CPUs by sysfs
oleg132-server: MODOPTS_LIBCFS=
oleg132-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
Starting client oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
Started clients oleg132-client.virtnet: 
192.168.201.132@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8e328909e000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8e328909e000.idle_timeout=debug
disable quota as required
preparing... Tue Jun 23 11:03:02 EDT 2026
creating 0 files on mds1
striped dir -i0 -c1 -H all_char  /mnt/lustre/d10a.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771313 s, 13.6 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H all_char  /mnt/lustre/d10a.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141431 s, 7.4 MB/s
prepared Tue Jun 23 11:03:16 EDT 2026.
stop mds1
stop mds2
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
starting mds with OI scrub disabled (1)
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
fail_val=1
fail_loc=0x190
192.168.201.132@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg132-client.virtnet /mnt/lustre (opts:)
stopping mds1
stopping mds2
starting MDTs with OI scrub disabled (2)
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
stopping mds1
stopping mds2
starting MDTs without disabling OI scrub
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
fail_loc=0
fail_val=0
PASS 10a (614s)

== sanity-scrub test 11: OI scrub skips the new created objects only once ========================================================== 11:08:12 (1782227292)
Checking servers environments
Checking clients oleg132-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg132-server'
oleg132-server: oleg132-server.virtnet: executing load_modules_local
oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg132-server: Loading modules from /home/green/git/lustre-release/lustre
oleg132-server: detected 4 online CPUs by sysfs
oleg132-server: MODOPTS_LIBCFS=
oleg132-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
Starting client oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
Started clients oleg132-client.virtnet: 
192.168.201.132@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8e3290de5800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8e3290de5800.idle_timeout=debug
disable quota as required
striped dir -i0 -c1 -H crush2  /mnt/lustre/d11.sanity-scrub/mds1
total: 100 open/close in 1.87 seconds: 53.48 ops/second
striped dir -i1 -c1 -H all_char  /mnt/lustre/d11.sanity-scrub/mds2
total: 100 open/close in 1.44 seconds: 69.46 ops/second
Started LFSCK on the device lustre-MDT0000: scrub
Started LFSCK on the device lustre-MDT0001: scrub
Started LFSCK on the device lustre-MDT0000: scrub
Started LFSCK on the device lustre-MDT0001: scrub
PASS 11 (126s)

== sanity-scrub test 12: OI scrub can rebuild invalid /O entries ========================================================== 11:10:18 (1782227418)
fail_loc=0x195
 - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 641 prealloc_next_id: 596 count: 46
total: 78 open/close in 0.95 seconds: 82.38 ops/second
192.168.201.132@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg132-client.virtnet /mnt/lustre (opts:)
Stopping /mnt/lustre-ost1 (opts:) on oleg132-server
fail_loc=0x233
Start ost1: mount -t lustre -o localrecov -o user_xattr,noscrub  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Started lustre-OST0000
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
Started LFSCK on the device lustre-OST0000: scrub
fail_loc=0
PASS 12 (238s)

== sanity-scrub test 13: OI scrub can rebuild missed /O entries ========================================================== 11:14:17 (1782227657)
fail_loc=0x196
 - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 705 prealloc_next_id: 674 count: 32
total: 64 open/close in 1.94 seconds: 32.95 ops/second
fail_loc=0
192.168.201.132@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg132-client.virtnet /mnt/lustre (opts:)
Stopping /mnt/lustre-ost1 (opts:) on oleg132-server
Start ost1: mount -t lustre -o localrecov -o user_xattr,noscrub  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Started lustre-OST0000
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre