-----============= acceptance-small: sanity-scrub ============----- Thu Apr  9 15:37:17 EDT 2026
mgs: Rocky Linux release 8.10 (Green Obsidian)
MGS_OS_ID_LIKE=rhel centos fedora rocky
MGS_OS_VERSION_ID=8.10
MGS_OS_ID=rocky
MGS_OS_VERSION_CODE=134873088
mds1: Rocky Linux release 8.10 (Green Obsidian)
MDS1_OS_VERSION_ID=8.10
MDS1_OS_VERSION_CODE=134873088
MDS1_OS_ID_LIKE=rhel centos fedora rocky
MDS1_OS_ID=rocky
ost1: Rocky Linux release 8.10 (Green Obsidian)
OST1_OS_VERSION_CODE=134873088
OST1_OS_ID_LIKE=rhel centos fedora rocky
OST1_OS_VERSION_ID=8.10
OST1_OS_ID=rocky
client: Rocky Linux release 8.10 (Green Obsidian)
CLIENT_OS_ID=rocky
CLIENT_OS_VERSION_CODE=134873088
CLIENT_OS_VERSION_ID=8.10
CLIENT_OS_ID_LIKE=rhel centos fedora rocky
oleg205-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-scrub.*ex': No such file or directory
excepting tests: 
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg205-server'
oleg205-server: oleg205-server.virtnet: executing load_modules_local
oleg205-server: Loading modules from /home/green/git/lustre-release/lustre
oleg205-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg205-server: detected 4 online CPUs by sysfs
oleg205-server: MODOPTS_LIBCFS=
oleg205-server: Force libcfs to create 2 CPU partitions
client=34681676 MDS=34681676 OSS=34681676
Stopping clients: oleg205-client.virtnet /mnt/lustre (opts:)
Stopping client oleg205-client.virtnet /mnt/lustre opts:
Stopping clients: oleg205-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg205-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg205-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg205-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg205-server

unloading modules via unload_modules_local on: 'oleg205-server'
oleg205-server: oleg205-server.virtnet: executing unload_modules_local
oleg205-server: 
modules unloaded.
=== sanity-scrub: start setup 15:39:04 (1775763544) ===
Stopping clients: oleg205-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg205-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 2
oleg205-server: oleg205-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg205-server'
oleg205-server: oleg205-server.virtnet: executing load_modules_local
oleg205-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg205-server: Loading modules from /home/green/git/lustre-release/lustre
oleg205-server: detected 4 online CPUs by sysfs
oleg205-server: MODOPTS_LIBCFS=
oleg205-server: Force libcfs to create 2 CPU partitions
oleg205-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg205-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg205-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Formatting mgs, mds, osts
Format mds1: /dev/vdc
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
Format mds2: /dev/vdd
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
Format ost1: /dev/vde
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
Format ost2: /dev/vdf
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
Checking servers environments
Checking clients oleg205-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg205-server'
oleg205-server: oleg205-server.virtnet: executing load_modules_local
oleg205-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg205-server: Loading modules from /home/green/git/lustre-release/lustre
oleg205-server: detected 4 online CPUs by sysfs
oleg205-server: MODOPTS_LIBCFS=
oleg205-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
Commit the device label on /dev/vdc
Started lustre-MDT0000
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
Commit the device label on /dev/vdd
Started lustre-MDT0001
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
Commit the device label on /dev/vde
Started lustre-OST0000
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
Commit the device label on /dev/vdf
Started lustre-OST0001
Starting client: oleg205-client.virtnet:  -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre
Starting client oleg205-client.virtnet:  -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre
Started clients oleg205-client.virtnet: 
192.168.202.105@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9d819116c800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9d819116c800.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
disable quota as required
osd-ldiskfs.track_declares_assert=1
=== sanity-scrub: finish setup 15:42:48 (1775763768) ===

== sanity-scrub test 0: Do not auto trigger OI scrub for non-backup/restore case ========================================================== 15:42:50 (1775763770)
preparing... Thu Apr  9 15:42:53 EDT 2026
creating 0 files on mds1
striped dir -i0 -c1 -H all_char /mnt/lustre/d0.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287967 s, 36.4 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H crush /mnt/lustre/d0.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360956 s, 29.0 MB/s
prepared Thu Apr  9 15:43:00 EDT 2026.
stop mds1
stop mds2
starting MDTs without disabling OI scrub
oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
Starting client: oleg205-client.virtnet:  -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre
PASS 0 (83s)

== sanity-scrub test 1a: Auto trigger initial OI scrub when server mounts ========================================================== 15:44:13 (1775763853)
preparing... Thu Apr  9 15:44:17 EDT 2026
creating 0 files on mds1
striped dir -i0 -c1 -H crush2 /mnt/lustre/d1a.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432322 s, 24.3 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H all_char /mnt/lustre/d1a.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038033 s, 27.6 MB/s
prepared Thu Apr  9 15:44:26 EDT 2026.
stop mds1
stop mds2
start mds1 without disabling OI scrub: -o localrecov -o user_xattr
oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
Starting client: oleg205-client.virtnet:  -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre
fail_loc=0x193
fail_loc=0
192.168.202.105@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg205-client.virtnet /mnt/lustre (opts:)
stop mds1
start mds1 with disabling OI scrub: -o localrecov -o user_xattr,noscrub
oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
PASS 1a (110s)

== sanity-scrub test 1b: Trigger OI scrub when MDT mounts for OI files remove/recreate case ========================================================== 15:46:03 (1775763963)
Checking servers environments
Checking clients oleg205-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg205-server'
oleg205-server: oleg205-server.virtnet: executing load_modules_local
oleg205-server: Loading modules from /home/green/git/lustre-release/lustre
oleg205-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg205-server: detected 4 online CPUs by sysfs
oleg205-server: MODOPTS_LIBCFS=
oleg205-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg205-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg205-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg205-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg205-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg205-client.virtnet:  -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre
Starting client oleg205-client.virtnet:  -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre
Started clients oleg205-client.virtnet: 
192.168.202.105@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9d8188461000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9d8188461000.idle_timeout=debug
disable quota as required
preparing... Thu Apr  9 15:47:16 EDT 2026
creating 0 files on mds1
striped dir -i0 -c1 -H crush /mnt/lustre/d1b.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561808 s, 18.7 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H crush /mnt/lustre/d1b.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548283 s, 19.1 MB/s
prepared Thu Apr  9 15:47:26 EDT 2026.
fail_loc=0x198
fail_loc=0
stop mds1
stop mds2
removing OI files on mds1: idx=
removed '/mnt/lustre-brpt/oi.16.0'
removed '/mnt/lustre-brpt/oi.16.1'
removed '/mnt/lustre-brpt/oi.16.10'
removed '/mnt/lustre-brpt/oi.16.11'
removed '/mnt/lustre-brpt/oi.16.12'
removed '/mnt/lustre-brpt/oi.16.13'
removed '/mnt/lustre-brpt/oi.16.14'
removed '/mnt/lustre-brpt/oi.16.15'
removed '/mnt/lustre-brpt/oi.16.16'
removed '/mnt/lustre-brpt/oi.16.17'
removed '/mnt/lustre-brpt/oi.16.18'
removed '/mnt/lustre-brpt/oi.16.19'
removed '/mnt/lustre-brpt/oi.16.2'
removed '/mnt/lustre-brpt/oi.16.20'
removed '/mnt/lustre-brpt/oi.16.21'
removed '/mnt/lustre-brpt/oi.16.22'
removed '/mnt/lustre-brpt/oi.16.23'
removed '/mnt/lustre-brpt/oi.16.24'
removed '/mnt/lustre-brpt/oi.16.25'
removed '/mnt/lustre-brpt/oi.16.26'
removed '/mnt/lustre-brpt/oi.16.27'
removed '/mnt/lustre-brpt/oi.16.28'
removed '/mnt/lustre-brpt/oi.16.29'
removed '/mnt/lustre-brpt/oi.16.3'
removed '/mnt/lustre-brpt/oi.16.30'
removed '/mnt/lustre-brpt/oi.16.31'
removed '/mnt/lustre-brpt/oi.16.32'
removed '/mnt/lustre-brpt/oi.16.33'
removed '/mnt/lustre-brpt/oi.16.34'
removed '/mnt/lustre-brpt/oi.16.35'
removed '/mnt/lustre-brpt/oi.16.36'
removed '/mnt/lustre-brpt/oi.16.37'
removed '/mnt/lustre-brpt/oi.16.38'
removed '/mnt/lustre-brpt/oi.16.39'
removed '/mnt/lustre-brpt/oi.16.4'
removed '/mnt/lustre-brpt/oi.16.40'
removed '/mnt/lustre-brpt/oi.16.41'
removed '/mnt/lustre-brpt/oi.16.42'
removed '/mnt/lustre-brpt/oi.16.43'
removed '/mnt/lustre-brpt/oi.16.44'
removed '/mnt/lustre-brpt/oi.16.45'
removed '/mnt/lustre-brpt/oi.16.46'
removed '/mnt/lustre-brpt/oi.16.47'
removed '/mnt/lustre-brpt/oi.16.48'
removed '/mnt/lustre-brpt/oi.16.49'
removed '/mnt/lustre-brpt/oi.16.5'
removed '/mnt/lustre-brpt/oi.16.50'
removed '/mnt/lustre-brpt/oi.16.51'
removed '/mnt/lustre-brpt/oi.16.52'
removed '/mnt/lustre-brpt/oi.16.53'
removed '/mnt/lustre-brpt/oi.16.54'
removed '/mnt/lustre-brpt/oi.16.55'
removed '/mnt/lustre-brpt/oi.16.56'
removed '/mnt/lustre-brpt/oi.16.57'
removed '/mnt/lustre-brpt/oi.16.58'
removed '/mnt/lustre-brpt/oi.16.59'
removed '/mnt/lustre-brpt/oi.16.6'
removed '/mnt/lustre-brpt/oi.16.60'
removed '/mnt/lustre-brpt/oi.16.61'
removed '/mnt/lustre-brpt/oi.16.62'
removed '/mnt/lustre-brpt/oi.16.63'
removed '/mnt/lustre-brpt/oi.16.7'
removed '/mnt/lustre-brpt/oi.16.8'
removed '/mnt/lustre-brpt/oi.16.9'
removing OI files on mds2: idx=
removed '/mnt/lustre-brpt/oi.16.0'
removed '/mnt/lustre-brpt/oi.16.1'
removed '/mnt/lustre-brpt/oi.16.10'
removed '/mnt/lustre-brpt/oi.16.11'
removed '/mnt/lustre-brpt/oi.16.12'
removed '/mnt/lustre-brpt/oi.16.13'
removed '/mnt/lustre-brpt/oi.16.14'
removed '/mnt/lustre-brpt/oi.16.15'
removed '/mnt/lustre-brpt/oi.16.16'
removed '/mnt/lustre-brpt/oi.16.17'
removed '/mnt/lustre-brpt/oi.16.18'
removed '/mnt/lustre-brpt/oi.16.19'
removed '/mnt/lustre-brpt/oi.16.2'
removed '/mnt/lustre-brpt/oi.16.20'
removed '/mnt/lustre-brpt/oi.16.21'
removed '/mnt/lustre-brpt/oi.16.22'
removed '/mnt/lustre-brpt/oi.16.23'
removed '/mnt/lustre-brpt/oi.16.24'
removed '/mnt/lustre-brpt/oi.16.25'
removed '/mnt/lustre-brpt/oi.16.26'
removed '/mnt/lustre-brpt/oi.16.27'
removed '/mnt/lustre-brpt/oi.16.28'
removed '/mnt/lustre-brpt/oi.16.29'
removed '/mnt/lustre-brpt/oi.16.3'
removed '/mnt/lustre-brpt/oi.16.30'
removed '/mnt/lustre-brpt/oi.16.31'
removed '/mnt/lustre-brpt/oi.16.32'
removed '/mnt/lustre-brpt/oi.16.33'
removed '/mnt/lustre-brpt/oi.16.34'
removed '/mnt/lustre-brpt/oi.16.35'
removed '/mnt/lustre-brpt/oi.16.36'
removed '/mnt/lustre-brpt/oi.16.37'
removed '/mnt/lustre-brpt/oi.16.38'
removed '/mnt/lustre-brpt/oi.16.39'
removed '/mnt/lustre-brpt/oi.16.4'
removed '/mnt/lustre-brpt/oi.16.40'
removed '/mnt/lustre-brpt/oi.16.41'
removed '/mnt/lustre-brpt/oi.16.42'
removed '/mnt/lustre-brpt/oi.16.43'
removed '/mnt/lustre-brpt/oi.16.44'
removed '/mnt/lustre-brpt/oi.16.45'
removed '/mnt/lustre-brpt/oi.16.46'
removed '/mnt/lustre-brpt/oi.16.47'
removed '/mnt/lustre-brpt/oi.16.48'
removed '/mnt/lustre-brpt/oi.16.49'
removed '/mnt/lustre-brpt/oi.16.5'
removed '/mnt/lustre-brpt/oi.16.50'
removed '/mnt/lustre-brpt/oi.16.51'
removed '/mnt/lustre-brpt/oi.16.52'
removed '/mnt/lustre-brpt/oi.16.53'
removed '/mnt/lustre-brpt/oi.16.54'
removed '/mnt/lustre-brpt/oi.16.55'
removed '/mnt/lustre-brpt/oi.16.56'
removed '/mnt/lustre-brpt/oi.16.57'
removed '/mnt/lustre-brpt/oi.16.58'
removed '/mnt/lustre-brpt/oi.16.59'
removed '/mnt/lustre-brpt/oi.16.6'
removed '/mnt/lustre-brpt/oi.16.60'
removed '/mnt/lustre-brpt/oi.16.61'
removed '/mnt/lustre-brpt/oi.16.62'
removed '/mnt/lustre-brpt/oi.16.63'
removed '/mnt/lustre-brpt/oi.16.7'
removed '/mnt/lustre-brpt/oi.16.8'
removed '/mnt/lustre-brpt/oi.16.9'
start MDTs without disabling OI scrub
oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
Starting client: oleg205-client.virtnet:  -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre
PASS 1b (163s)

== sanity-scrub test 1c: Auto detect kinds of OI file(s) removed/recreated cases ========================================================== 15:48:47 (1775764127)
preparing... Thu Apr  9 15:48:51 EDT 2026
creating 0 files on mds1
striped dir -i0 -c1 -H crush /mnt/lustre/d1c.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296369 s, 35.4 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H crush /mnt/lustre/d1c.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081801 s, 12.8 MB/s
prepared Thu Apr  9 15:49:01 EDT 2026.
stop mds1
stop mds2
removing OI files on mds1: idx=0
removed '/mnt/lustre-brpt/oi.16.0'
removing OI files on mds2: idx=0
removed '/mnt/lustre-brpt/oi.16.0'
start MDTs with OI scrub disabled
oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
Started LFSCK on the device lustre-MDT0000: scrub
Started LFSCK on the device lustre-MDT0001: scrub
Checking servers environments
Checking clients oleg205-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg205-server'
oleg205-server: oleg205-server.virtnet: executing load_modules_local
oleg205-server: Loading modules from /home/green/git/lustre-release/lustre
oleg205-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg205-server: detected 4 online CPUs by sysfs
oleg205-server: MODOPTS_LIBCFS=
oleg205-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg205-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg205-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg205-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg205-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg205-client.virtnet:  -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre
Starting client oleg205-client.virtnet:  -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre
Started clients oleg205-client.virtnet: 
192.168.202.105@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9d8199106000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9d8199106000.idle_timeout=debug
disable quota as required
preparing... Thu Apr  9 15:51:37 EDT 2026
creating 0 files on mds1
striped dir -i0 -c1 -H all_char /mnt/lustre/d1c.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328354 s, 31.9 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H fnv_1a_64 /mnt/lustre/d1c.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487918 s, 21.5 MB/s
prepared Thu Apr  9 15:51:46 EDT 2026.
stop mds1
stop mds2
removing OI files on mds1: idx=2
removed '/mnt/lustre-brpt/oi.16.2'
removed '/mnt/lustre-brpt/oi.16.4'
removed '/mnt/lustre-brpt/oi.16.8'
removed '/mnt/lustre-brpt/oi.16.16'
removed '/mnt/lustre-brpt/oi.16.32'
removing OI files on mds2: idx=2
removed '/mnt/lustre-brpt/oi.16.2'
removed '/mnt/lustre-brpt/oi.16.4'
removed '/mnt/lustre-brpt/oi.16.8'
removed '/mnt/lustre-brpt/oi.16.16'
removed '/mnt/lustre-brpt/oi.16.32'
start MDTs with OI scrub disabled
oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
Started LFSCK on the device lustre-MDT0000: scrub
Started LFSCK on the device lustre-MDT0001: scrub
Checking servers environments
Checking clients oleg205-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg205-server'
oleg205-server: oleg205-server.virtnet: executing load_modules_local
oleg205-server: Loading modules from /home/green/git/lustre-release/lustre
oleg205-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg205-server: detected 4 online CPUs by sysfs
oleg205-server: MODOPTS_LIBCFS=
oleg205-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg205-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg205-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg205-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg205-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg205-client.virtnet:  -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre
Starting client oleg205-client.virtnet:  -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre
Started clients oleg205-client.virtnet: 
192.168.202.105@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9d819024e800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9d819024e800.idle_timeout=debug
disable quota as required
preparing... Thu Apr  9 15:54:41 EDT 2026
creating 0 files on mds1
striped dir -i0 -c1 -H all_char /mnt/lustre/d1c.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871124 s, 12.0 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H crush /mnt/lustre/d1c.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343442 s, 30.5 MB/s
prepared Thu Apr  9 15:54:51 EDT 2026.
stop mds1
stop mds2
removing OI files on mds1: idx=3
removed '/mnt/lustre-brpt/oi.16.3'
removed '/mnt/lustre-brpt/oi.16.9'
removed '/mnt/lustre-brpt/oi.16.27'
removing OI files on mds2: idx=3
removed '/mnt/lustre-brpt/oi.16.3'
removed '/mnt/lustre-brpt/oi.16.9'
removed '/mnt/lustre-brpt/oi.16.27'
start MDTs with OI scrub disabled
oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
Started LFSCK on the device lustre-MDT0000: scrub
Started LFSCK on the device lustre-MDT0001: scrub
PASS 1c (463s)

== sanity-scrub test 2: Trigger OI scrub when MDT mounts for backup/restore case ========================================================== 15:56:30 (1775764590)
Checking servers environments
Checking clients oleg205-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg205-server'
oleg205-server: oleg205-server.virtnet: executing load_modules_local
oleg205-server: Loading modules from /home/green/git/lustre-release/lustre
oleg205-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg205-server: detected 4 online CPUs by sysfs
oleg205-server: MODOPTS_LIBCFS=
oleg205-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg205-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg205-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg205-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg205-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg205-client.virtnet:  -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre
Starting client oleg205-client.virtnet:  -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre
Started clients oleg205-client.virtnet: 
192.168.202.105@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9d81892c7000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9d81892c7000.idle_timeout=debug
disable quota as required
preparing... Thu Apr  9 15:57:43 EDT 2026
creating 0 files on mds1
striped dir -i0 -c1 -H all_char /mnt/lustre/d2.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423178 s, 24.8 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H crush /mnt/lustre/d2.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499573 s, 21.0 MB/s
prepared Thu Apr  9 15:57:53 EDT 2026.
stop mds1
stop mds2
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
starting MDTs without disabling OI scrub
oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
Starting client: oleg205-client.virtnet:  -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre
PASS 2 (202s)

== sanity-scrub test 4a: Auto trigger OI scrub if bad OI mapping was found (1) ========================================================== 15:59:51 (1775764791)
preparing... Thu Apr  9 15:59:56 EDT 2026
creating 0 files on mds1
striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d4a.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279507 s, 37.5 MB/s
creating 0 files on mds2
striped dir -i1 -c1 -H all_char /mnt/lustre/d4a.sanity-scrub/mds2
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294911 s, 35.6 MB/s
prepared Thu Apr  9 16:00:04 EDT 2026.
stop mds1
stop mds2
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
file-level backup/restore on mds2:/dev/mapper/mds2_flakey
backup data
reformat new device
Format mds2: /dev/mapper/mds2_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
starting MDTs with OI scrub disabled
oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1
Starting client: oleg205-client.virtnet:  -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre
PASS 4a (129s)

== sanity-scrub test 4b: Auto trigger OI scrub if bad OI mapping was found (2) ========================================================== 16:02:01 (1775764921)
preparing... Thu Apr  9 16:02:02 EDT 2026
creating 5 files on mds1
striped dir -i0 -c1 -H crush /mnt/lustre/d4b.sanity-scrub/mds1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140562 s, 74.6 MB/s
creating 5 files on mds2
striped dir -i1 -c1 -H crush /mnt/lustre/d4b.sanity-scrub/mds2