-----============= acceptance-small: sanity-lfsck ============----- Tue Sep  8 07:22:58 EDT 2026
MGS_OS_ID_LIKE=rhel centos fedora rocky
MGS_OS_VERSION_ID=8.10
MGS_OS_ID=rocky
MGS_OS_VERSION_CODE=134873088
MDS1_OS_VERSION_ID=8.10
MDS1_OS_VERSION_CODE=134873088
MDS1_OS_ID_LIKE=rhel centos fedora rocky
MDS1_OS_ID=rocky
OST1_OS_VERSION_CODE=134873088
OST1_OS_ID_LIKE=rhel centos fedora rocky
OST1_OS_VERSION_ID=8.10
OST1_OS_ID=rocky
CLIENT_OS_ID=rocky
CLIENT_OS_VERSION_CODE=134873088
CLIENT_OS_VERSION_ID=8.10
CLIENT_OS_ID_LIKE=rhel centos fedora rocky
oleg220-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-lfsck.*ex': No such file or directory
excepting tests: 18b 23b
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg220-server'
oleg220-server: oleg220-server.virtnet: executing load_modules_local
oleg220-server: kmemleak disabled
oleg220-server: Loading modules from /home/green/git/lustre-release/lustre
oleg220-server: detected 4 online CPUs by sysfs
oleg220-server: MODOPTS_LIBCFS=
oleg220-server: Force libcfs to create 2 CPU partitions
oleg220-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
client=34683240 MDS=34683240 OSS=34683240
Stopping clients: oleg220-client.virtnet /mnt/lustre (opts:)
Stopping client oleg220-client.virtnet /mnt/lustre opts:
Stopping clients: oleg220-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg220-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg220-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg220-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg220-server
unloading modules via unload_modules_local on: 'oleg220-server'
oleg220-server: oleg220-server.virtnet: executing unload_modules_local
oleg220-server: kmemleak disabled
modules unloaded.
=== sanity-lfsck: start setup 07:24:50 (1788866690) ===
Stopping clients: oleg220-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg220-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 2
oleg220-server: oleg220-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1754936 Sep  8 04:31 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg220-server'
oleg220-server: oleg220-server.virtnet: executing load_modules_local
oleg220-server: kmemleak disabled
oleg220-server: Loading modules from /home/green/git/lustre-release/lustre
oleg220-server: detected 4 online CPUs by sysfs
oleg220-server: MODOPTS_LIBCFS=
oleg220-server: Force libcfs to create 2 CPU partitions
oleg220-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg220-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg220-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg220-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/vdc
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Format mds2: /dev/vdd
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Format ost1: /dev/vde
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Format ost2: /dev/vdf
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Checking servers environments
Checking clients oleg220-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg220-server'
oleg220-server: oleg220-server.virtnet: executing load_modules_local
oleg220-server: kmemleak disabled
oleg220-server: Loading modules from /home/green/git/lustre-release/lustre
oleg220-server: detected 4 online CPUs by sysfs
oleg220-server: MODOPTS_LIBCFS=
oleg220-server: Force libcfs to create 2 CPU partitions
oleg220-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Setup mgs, mdt, osts
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Commit the device label on /dev/vdc
Started lustre-MDT0000
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Commit the device label on /dev/vdd
Started lustre-MDT0001
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Commit the device label on /dev/vde
Started lustre-OST0000
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Commit the device label on /dev/vdf
Started lustre-OST0001
Starting client: oleg220-client.virtnet:  -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre
Starting client oleg220-client.virtnet:  -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre
Started clients oleg220-client.virtnet: 
192.168.202.120@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8d2146006800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8d2146006800.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
disable quota as required
osd-ldiskfs.track_declares_assert=1
=== sanity-lfsck: finish setup 07:28:28 (1788866908) ===

== sanity-lfsck test 0: Control LFSCK manually =========== 07:28:33 (1788866913)
preparing... 3 * 3 files will be created Tue Sep  8 07:28:35 EDT 2026.
total: 3 mkdir in 0.02 seconds: 130.75 ops/second
total: 3 create in 0.03 seconds: 99.45 ops/second
total: 3 mkdir in 0.11 seconds: 26.51 ops/second
prepared Tue Sep  8 07:28:43 EDT 2026.
fail_val=3
fail_loc=0x1600
Started LFSCK on the device lustre-MDT0000: scrub namespace
name: lfsck_namespace
magic: 0xa06249ff
version: 2
status: scanning-phase1
flags:
param:
last_completed_time: N/A
time_since_last_completed: N/A
latest_start_time: 1788866925
time_since_latest_start: 1 seconds
last_checkpoint_time: N/A
time_since_last_checkpoint: N/A
latest_start_position: 13, N/A, N/A
last_checkpoint_position: N/A, N/A, N/A
first_failure_position: N/A, N/A, N/A
checked_phase1: 0
checked_phase2: 0
updated_phase1: 0
updated_phase2: 0
failed_phase1: 0
failed_phase2: 0
directories: 0
dirent_repaired: 0
linkea_repaired: 0
nlinks_repaired: 0
multiple_linked_checked: 0
multiple_linked_repaired: 0
unknown_inconsistency: 0
unmatched_pairs_repaired: 0
dangling_repaired: 0
multiple_referenced_repaired: 0
bad_file_type_repaired: 0
lost_dirent_repaired: 0
local_lost_found_scanned: 0
local_lost_found_moved: 0
local_lost_found_skipped: 0
local_lost_found_failed: 0
striped_dirs_scanned: 0
striped_dirs_repaired: 0
striped_dirs_failed: 0
striped_dirs_disabled: 0
striped_dirs_skipped: 0
striped_shards_scanned: 0
striped_shards_repaired: 0
striped_shards_failed: 0
striped_shards_skipped: 0
name_hash_repaired: 0
linkea_overflow_cleared: 0
agent_entries_repaired: 0
success_count: 0
run_time_phase1: 1 seconds
run_time_phase2: 0 seconds
average_speed_phase1: 0 items/sec
average_speed_phase2: N/A
average_speed_total: 0 items/sec
real_time_speed_phase1: 0 items/sec
real_time_speed_phase2: N/A
current_position: 12, N/A, N/A
Stopped LFSCK on the device lustre-MDT0000.
Started LFSCK on the device lustre-MDT0000: scrub namespace
fail_loc=0
fail_val=0
Waiting 32s for 'completed'
Updated after 2s: want 'completed' got 'completed'
Started LFSCK on the device lustre-MDT0000: scrub namespace
stopall, should NOT crash LU-3649
Stopping clients: oleg220-client.virtnet /mnt/lustre (opts:)
Stopping client oleg220-client.virtnet /mnt/lustre opts:
Stopping clients: oleg220-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg220-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg220-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg220-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg220-server
PASS 0 (79s)

== sanity-lfsck test 1a: LFSCK can find out and repair crashed FID-in-dirent ========================================================== 07:29:53 (1788866993)
Checking servers environments
Checking clients oleg220-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg220-server'
oleg220-server: oleg220-server.virtnet: executing load_modules_local
oleg220-server: kmemleak disabled
oleg220-server: Loading modules from /home/green/git/lustre-release/lustre
oleg220-server: detected 4 online CPUs by sysfs
oleg220-server: MODOPTS_LIBCFS=
oleg220-server: Force libcfs to create 2 CPU partitions
oleg220-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Started lustre-MDT0000
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Started lustre-MDT0001
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Started lustre-OST0001
Starting client: oleg220-client.virtnet:  -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre
Starting client oleg220-client.virtnet:  -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre
Started clients oleg220-client.virtnet: 
192.168.202.120@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8d21517e7800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8d21517e7800.idle_timeout=debug
disable quota as required
preparing... 1 * 1 files will be created Tue Sep  8 07:31:35 EDT 2026.
total: 1 mkdir in 0.01 seconds: 75.80 ops/second
total: 1 create in 0.02 seconds: 47.80 ops/second
total: 1 mkdir in 0.01 seconds: 124.53 ops/second
prepared Tue Sep  8 07:31:40 EDT 2026.
fail_loc=0x1501
fail_loc=0
192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg220-client.virtnet /mnt/lustre (opts:)
Started LFSCK on the device lustre-MDT0000: scrub namespace
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Starting client: oleg220-client.virtnet:  -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre
fail_loc=0x1505
fail_loc=0
PASS 1a (146s)

== sanity-lfsck test 1b: LFSCK can find out and repair the missing FID-in-LMA ========================================================== 07:32:19 (1788867139)
preparing... 1 * 1 files will be created Tue Sep  8 07:32:23 EDT 2026.
total: 1 mkdir in 0.01 seconds: 123.62 ops/second
total: 1 create in 0.01 seconds: 141.58 ops/second
total: 1 mkdir in 0.02 seconds: 52.63 ops/second
prepared Tue Sep  8 07:32:27 EDT 2026.
fail_loc=0x1502
fail_loc=0
192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg220-client.virtnet /mnt/lustre (opts:)
fail_loc=0x1506
Started LFSCK on the device lustre-MDT0000: scrub namespace
fail_loc=0
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Starting client: oleg220-client.virtnet:  -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre
fail_loc=0x1505
fail_loc=0
PASS 1b (53s)

== sanity-lfsck test 1c: LFSCK can find out and repair lost FID-in-dirent ========================================================== 07:33:13 (1788867193)
preparing... 1 * 1 files will be created Tue Sep  8 07:33:17 EDT 2026.
total: 1 mkdir in 0.03 seconds: 35.24 ops/second
total: 1 create in 0.01 seconds: 118.30 ops/second
total: 1 mkdir in 0.01 seconds: 80.01 ops/second
prepared Tue Sep  8 07:33:22 EDT 2026.
fail_loc=0x1504
fail_loc=0
192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg220-client.virtnet /mnt/lustre (opts:)
Started LFSCK on the device lustre-MDT0000: scrub namespace
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Starting client: oleg220-client.virtnet:  -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre
fail_loc=0x1505
fail_loc=0
PASS 1c (51s)

== sanity-lfsck test 2a: LFSCK can find out and repair crashed linkEA entry ========================================================== 07:34:03 (1788867243)
preparing... 1 * 1 files will be created Tue Sep  8 07:34:07 EDT 2026.
total: 1 mkdir in 0.01 seconds: 133.44 ops/second
total: 1 create in 0.01 seconds: 121.40 ops/second
total: 1 mkdir in 0.01 seconds: 131.32 ops/second
prepared Tue Sep  8 07:34:12 EDT 2026.
fail_loc=0x1603
fail_loc=0
192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg220-client.virtnet /mnt/lustre (opts:)
Started LFSCK on the device lustre-MDT0000: scrub namespace
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Starting client: oleg220-client.virtnet:  -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre
PASS 2a (48s)

== sanity-lfsck test 2b: LFSCK can find out and remove invalid linkEA entry ========================================================== 07:34:51 (1788867291)
preparing... 1 * 1 files will be created Tue Sep  8 07:34:56 EDT 2026.
total: 1 mkdir in 0.01 seconds: 111.93 ops/second
total: 1 create in 0.01 seconds: 124.19 ops/second
total: 1 mkdir in 0.01 seconds: 82.28 ops/second
prepared Tue Sep  8 07:35:01 EDT 2026.
fail_loc=0x1604
fail_loc=0
192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg220-client.virtnet /mnt/lustre (opts:)
Started LFSCK on the device lustre-MDT0000: scrub namespace
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Starting client: oleg220-client.virtnet:  -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre
PASS 2b (50s)

== sanity-lfsck test 2c: LFSCK can find out and remove repeated linkEA entry ========================================================== 07:35:42 (1788867342)
preparing... 1 * 1 files will be created Tue Sep  8 07:35:46 EDT 2026.
total: 1 mkdir in 0.00 seconds: 220.07 ops/second
total: 1 create in 0.01 seconds: 124.25 ops/second
total: 1 mkdir in 0.03 seconds: 34.38 ops/second
prepared Tue Sep  8 07:35:51 EDT 2026.
fail_loc=0x1605
fail_loc=0
192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg220-client.virtnet /mnt/lustre (opts:)
Started LFSCK on the device lustre-MDT0000: scrub namespace
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Starting client: oleg220-client.virtnet:  -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre
PASS 2c (50s)

== sanity-lfsck test 2d: LFSCK can recover the missing linkEA entry ========================================================== 07:36:32 (1788867392)
preparing... 1 * 1 files will be created Tue Sep  8 07:36:36 EDT 2026.
total: 1 mkdir in 0.01 seconds: 171.76 ops/second
total: 1 create in 0.01 seconds: 175.10 ops/second
total: 1 mkdir in 0.01 seconds: 173.91 ops/second
prepared Tue Sep  8 07:36:40 EDT 2026.
fail_loc=0x161d
fail_loc=0
192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg220-client.virtnet /mnt/lustre (opts:)
Started LFSCK on the device lustre-MDT0000: scrub namespace
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Starting client: oleg220-client.virtnet:  -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre
PASS 2d (45s)

== sanity-lfsck test 2e: namespace LFSCK can verify remote object linkEA ========================================================== 07:37:17 (1788867437)
fail_loc=0x1603
fail_loc=0
Started LFSCK on the device lustre-MDT0000: scrub namespace
PASS 2e (16s)

== sanity-lfsck test 3: LFSCK can verify multiple-linked objects ========================================================== 07:37:33 (1788867453)
preparing... 4 * 4 files will be created Tue Sep  8 07:37:35 EDT 2026.
total: 4 mkdir in 0.06 seconds: 67.82 ops/second
total: 4 create in 0.04 seconds: 90.78 ops/second
total: 4 mkdir in 0.05 seconds: 74.13 ops/second
prepared Tue Sep  8 07:37:40 EDT 2026.
fail_loc=0x1603
fail_loc=0x1604
fail_loc=0
Started LFSCK on the device lustre-MDT0000: scrub namespace
PASS 3 (21s)

== sanity-lfsck test 4: FID-in-dirent can be rebuilt after MDT file-level backup/restore ========================================================== 07:37:54 (1788867474)
preparing... 3 * 3 files will be created Tue Sep  8 07:37:59 EDT 2026.
total: 3 mkdir in 0.03 seconds: 118.50 ops/second
total: 3 create in 0.03 seconds: 103.12 ops/second
total: 3 mkdir in 0.06 seconds: 51.10 ops/second
prepared Tue Sep  8 07:38:05 EDT 2026.
Stopping clients: oleg220-client.virtnet /mnt/lustre (opts:)
Stopping client oleg220-client.virtnet /mnt/lustre opts:
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
start mds1 with disabling OI scrub
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
fail_val=1
fail_loc=0x1601
Started LFSCK on the device lustre-MDT0000: scrub namespace
Waiting 32s for 'inconsistent'
Updated after 2s: want 'inconsistent' got 'inconsistent'
fail_loc=0
fail_val=0
Waiting 32s for 'completed'
Updated after 2s: want 'completed' got 'completed'
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Starting client: oleg220-client.virtnet:  -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre
fail_loc=0x1505
fail_loc=0
PASS 4 (138s)

== sanity-lfsck test 5: LFSCK can handle IGIF object upgrading ========================================================== 07:40:12 (1788867612)
preparing... 1 * 1 files will be created Tue Sep  8 07:40:16 EDT 2026.
fail_loc=0x1504
total: 1 mkdir in 0.03 seconds: 35.41 ops/second
total: 1 create in 0.03 seconds: 36.41 ops/second
total: 1 mkdir in 0.02 seconds: 43.85 ops/second
fail_loc=0
prepared Tue Sep  8 07:40:22 EDT 2026.
Stopping clients: oleg220-client.virtnet /mnt/lustre (opts:)
Stopping client oleg220-client.virtnet /mnt/lustre opts:
file-level backup/restore on mds1:/dev/mapper/mds1_flakey
backup data
reformat new device
Format mds1: /dev/mapper/mds1_flakey
restore data
remove recovery logs
removed '/mnt/lustre-brpt/CATALOGS'
start mds1 with disabling OI scrub
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
fail_val=1
fail_loc=0x1601
Started LFSCK on the device lustre-MDT0000: scrub namespace
Waiting 32s for 'inconsistent,upgrade'
Updated after 4s: want 'inconsistent,upgrade' got 'inconsistent,upgrade'
fail_loc=0
fail_val=0
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Starting client: oleg220-client.virtnet:  -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre
fail_loc=0x1505
fail_loc=0
PASS 5 (116s)

== sanity-lfsck test 6a: LFSCK resumes from last checkpoint (1) ========================================================== 07:42:08 (1788867728)
preparing... 5 * 5 files will be created Tue Sep  8 07:42:12 EDT 2026.
total: 5 mkdir in 0.03 seconds: 172.31 ops/second
total: 5 create in 0.03 seconds: 192.40 ops/second
total: 5 mkdir in 0.03 seconds: 151.13 ops/second
prepared Tue Sep  8 07:42:17 EDT 2026.
fail_val=1
fail_loc=0x1600
Started LFSCK on the device lustre-MDT0000: scrub namespace
fail_loc=0x80001608
fail_val=1
fail_loc=0x1600
Started LFSCK on the device lustre-MDT0000: scrub namespace
fail_loc=0
fail_val=0
PASS 6a (33s)

== sanity-lfsck test 6b: LFSCK resumes from last checkpoint (2) ========================================================== 07:42:41 (1788867761)
preparing... 5 * 5 files will be created Tue Sep  8 07:42:45 EDT 2026.
total: 5 mkdir in 0.08 seconds: 66.10 ops/second
total: 5 create in 0.04 seconds: 134.04 ops/second
total: 5 mkdir in 0.09 seconds: 56.41 ops/second
prepared Tue Sep  8 07:42:50 EDT 2026.
fail_val=1
fail_loc=0x1601
Started LFSCK on the device lustre-MDT0000: scrub namespace
fail_loc=0x80001609
fail_val=1
fail_loc=0x1601
Started LFSCK on the device lustre-MDT0000: scrub namespace
Additional debug for 6b
name: lfsck_namespace
magic: 0xa06249ff
version: 2
status: scanning-phase1
flags:
param:
last_completed_time: 1788867753
time_since_last_completed: 39 seconds
latest_start_time: 1788867786
time_since_latest_start: 6 seconds
last_checkpoint_time: 1788867780
time_since_last_checkpoint: 12 seconds
latest_start_position: 20010, [0x2000061c1:0x77:0x0], 0x3526fe12f174e84
last_checkpoint_position: 20008, [0x2000061c1:0x77:0x0], 0x1e13bb946ad9fc4
first_failure_position: N/A, N/A, N/A
checked_phase1: 10
checked_phase2: 0
updated_phase1: 0
updated_phase2: 0
failed_phase1: 0
failed_phase2: 0
directories: 2
dirent_repaired: 0
linkea_repaired: 0
nlinks_repaired: 0
multiple_linked_checked: 0
multiple_linked_repaired: 0
unknown_inconsistency: 0
unmatched_pairs_repaired: 0
dangling_repaired: 0
multiple_referenced_repaired: 0
bad_file_type_repaired: 0
lost_dirent_repaired: 0
local_lost_found_scanned: 0
local_lost_found_moved: 0
local_lost_found_skipped: 0
local_lost_found_failed: 0
striped_dirs_scanned: 0
striped_dirs_repaired: 0
striped_dirs_failed: 0
striped_dirs_disabled: 0
striped_dirs_skipped: 0
striped_shards_scanned: 0
striped_shards_repaired: 0
striped_shards_failed: 0
striped_shards_skipped: 0
name_hash_repaired: 0
linkea_overflow_cleared: 0
agent_entries_repaired: 0
success_count: 14
run_time_phase1: 14 seconds
run_time_phase2: 0 seconds
average_speed_phase1: 0 items/sec
average_speed_phase2: N/A
average_speed_total: 0 items/sec
real_time_speed_phase1: 0 items/sec
real_time_speed_phase2: N/A
current_position: 20009, [0x2000061c1:0x77:0x0], 0x89c5a2972006753
fail_loc=0
fail_val=0
PASS 6b (38s)

== sanity-lfsck test 7a: non-stopped LFSCK should auto restarts after MDS remount (1) ========================================================== 07:43:19 (1788867799)
preparing... 5 * 5 files will be created Tue Sep  8 07:43:23 EDT 2026.
total: 5 mkdir in 0.05 seconds: 95.66 ops/second
total: 5 create in 0.03 seconds: 156.52 ops/second
total: 5 mkdir in 0.03 seconds: 147.20 ops/second
prepared Tue Sep  8 07:43:28 EDT 2026.
192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg220-client.virtnet /mnt/lustre (opts:)
fail_val=1
fail_loc=0x1601
Started LFSCK on the device lustre-MDT0000: scrub namespace
stop mds1
fail_loc=0
fail_val=0
start mds1
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
PASS 7a (51s)

== sanity-lfsck test 7b: non-stopped LFSCK should auto restarts after MDS remount (2) ========================================================== 07:44:11 (1788867851)
Checking servers environments
Checking clients oleg220-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg220-server'
oleg220-server: oleg220-server.virtnet: executing load_modules_local
oleg220-server: kmemleak disabled
oleg220-server: Loading modules from /home/green/git/lustre-release/lustre
oleg220-server: detected 4 online CPUs by sysfs
oleg220-server: MODOPTS_LIBCFS=
oleg220-server: Force libcfs to create 2 CPU partitions
oleg220-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg220-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg220-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg220-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg220-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg220-client.virtnet:  -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre
Starting client oleg220-client.virtnet:  -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre
Started clients oleg220-client.virtnet: 
192.168.202.120@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8d2150dfc000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8d2150dfc000.idle_timeout=debug
disable quota as required
preparing... 2 * 2 files will be created Tue Sep  8 07:45:27 EDT 2026.
total: 2 mkdir in 0.02 seconds: 110.23 ops/second
total: 2 create in 0.01 seconds: 164.43 ops/second
total: 2 mkdir in 0.02 seconds: 99.14 ops/second
prepared Tue Sep  8 07:45:32 EDT 2026.
fail_loc=0x1604
fail_val=1
fail_loc=0x1602
Started LFSCK on the device lustre-MDT0000: scrub namespace
192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg220-client.virtnet /mnt/lustre (opts:)
stop mds1
fail_loc=0
fail_val=0
start mds1
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
PASS 7b (122s)

== sanity-lfsck test 8: LFSCK state machine ============== 07:46:13 (1788867973)
formatall
oleg220-server: oleg220-server.virtnet: executing set_hostid
oleg220-server: oleg220-server.virtnet: executing load_modules_local
setupall
oleg220-server: oleg220-server.virtnet: executing load_modules_local
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Using TIMEOUT=20
preparing... 20 * 20 files will be created Tue Sep  8 07:49:31 EDT 2026.
total: 20 mkdir in 0.18 seconds: 108.14 ops/second
total: 20 create in 0.23 seconds: 86.31 ops/second
total: 20 mkdir in 0.13 seconds: 158.40 ops/second
prepared Tue Sep  8 07:49:40 EDT 2026.
fail_loc=0x1603
fail_loc=0x1604
192.168.202.120@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg220-client.virtnet /mnt/lustre (opts:)
fail_val=2
fail_loc=0x1601
Started LFSCK on the device lustre-MDT0000: scrub namespace
Stopped LFSCK on the device lustre-MDT0000.
Started LFSCK on the device lustre-MDT0000: scrub namespace
fail_loc=0x80001609
Waiting 32s for 'failed'
Updated after 2s: want 'failed' got 'failed'
fail_loc=0x1600
Started LFSCK on the device lustre-MDT0000: scrub namespace
fail_loc=0x160a
stop mds1
fail_loc=0x160b
start mds1
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
fail_loc=0x1601
Started LFSCK on the device lustre-MDT0000: scrub namespace
stop mds1
fail_loc=0x160b
start mds1
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
stop mds1
start mds1 without resume LFSCK
oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
fail_val=2
fail_loc=0x1602
Started LFSCK on the device lustre-MDT0000: scrub namespace
Waiting 32s for 'scanned-once,inconsistent'
Updated after 4s: want 'scanned-once,inconsistent' got 'scanned-once,inconsistent'
fail_loc=0
fail_val=0
Waiting 32s for 'completed'
Updated after 2s: want 'completed' got 'completed'
PASS 8 (326s)

== sanity-lfsck test 9a: LFSCK speed control (1) ========= 07:51:39 (1788868299)
Checking servers environments
Checking clients oleg220-client.virtnet environments
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg220-server'
oleg220-server: oleg220-server.virtnet: executing load_modules_local
oleg220-server: kmemleak disabled
oleg220-server: Loading modules from /home/green/git/lustre-release/lustre
oleg220-server: detected 4 online CPUs by sysfs
oleg220-server: MODOPTS_LIBCFS=
oleg220-server: Force libcfs to create 2 CPU partitions
oleg220-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg220-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg220-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg220-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg220-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg220-client.virtnet:  -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre
Starting client oleg220-client.virtnet:  -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre
Started clients oleg220-client.virtnet: 
192.168.202.120@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff8d214876d800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff8d214876d800.idle_timeout=debug
disable quota as required
 - open/close 788 (time 1788868399.24 total 10.01 last 78.73)
 - open/close 1498 (time 1788868409.27 total 20.04 last 70.76)
 - open/close 2142 (time 1788868419.27 total 30.04 last 64.40)
 - open/close 2970 (time 1788868429.28 total 40.05 last 82.75)
 - open/close 3802 (time 1788868439.30 total 50.06 last 83.08)
 - open/close 4677 (time 1788868449.30 total 60.07 last 87.46)
total: 5000 open/close in 63.88 seconds: 78.27 ops/second
Started LFSCK on the device lustre-MDT0000: scrub layout
PASS 9a (190s)

== sanity-lfsck test 9b: LFSCK speed control (2) ========= 07:54:50 (1788868490)
preparing... 0 * 0 files will be created Tue Sep  8 07:55:38 EDT 2026.
prepared Tue Sep  8 07:55:42 EDT 2026.
Preparing another 50 * 50 files (with error) at Tue Sep  8 07:55:42 EDT 2026.
fail_loc=0x1604
total: 50 mkdir in 0.39 seconds: 127.24 ops/second
total: 50 create in 0.39 seconds: 128.26 ops/second
fail_loc=0x160c
Started LFSCK on the device lustre-MDT0000: scrub namespace
fail_loc=0
Prepared at Tue Sep  8 07:56:12 EDT 2026.
Started LFSCK on the device lustre-MDT0000: scrub namespace
PASS 9b (119s)

== sanity-lfsck test 10: System is available during LFSCK scanning ========================================================== 07:56:49 (1788868609)
preparing... 1 * 1 files will be created Tue Sep  8 07:57:33 EDT 2026.
total: 1 mkdir in 0.01 seconds: 147.34 ops/second
total: 1 create in 0.01 seconds: 158.03 ops/second
total: 1 mkdir in 0.00 seconds: 210.75 ops/second
prepared Tue Sep  8 07:57:37 EDT 2026.
Preparing more files with error at Tue Sep  8 07:57:37 EDT 2026.
fail_loc=0x1603
fail_loc=0x1604