-----============= acceptance-small: sanity-lfsck ============----- Thu Jan  1 19:57:48 EST 2026
mgs: Rocky Linux release 8.10 (Green Obsidian)
MGS_OS_ID_LIKE=rhel centos fedora rocky
MGS_OS_VERSION_ID=8.10
MGS_OS_ID=rocky
MGS_OS_VERSION_CODE=134873088
mds1: Rocky Linux release 8.10 (Green Obsidian)
MDS1_OS_VERSION_ID=8.10
MDS1_OS_VERSION_CODE=134873088
MDS1_OS_ID_LIKE=rhel centos fedora rocky
MDS1_OS_ID=rocky
ost1: Rocky Linux release 8.10 (Green Obsidian)
OST1_OS_VERSION_CODE=134873088
OST1_OS_ID_LIKE=rhel centos fedora rocky
OST1_OS_VERSION_ID=8.10
OST1_OS_ID=rocky
client: Rocky Linux release 8.10 (Green Obsidian)
CLIENT_OS_ID=rocky
CLIENT_OS_VERSION_CODE=134873088
CLIENT_OS_VERSION_ID=8.10
CLIENT_OS_ID_LIKE=rhel centos fedora rocky
oleg322-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-lfsck.*ex': No such file or directory
excepting tests: 18b 23b
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg322-server'
oleg322-server: oleg322-server.virtnet: executing load_modules_local
oleg322-server: Loading modules from /home/green/git/lustre-release/lustre
oleg322-server: detected 4 online CPUs by sysfs
oleg322-server: MODOPTS_LIBCFS=
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
oleg322-server: Force libcfs to create 2 CPU partitions
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
client=34668548 MDS=34668548 OSS=34668548
Stopping clients: oleg322-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg322-client.virtnet /mnt/lustre2 (opts:)
Stopping client oleg322-client.virtnet /mnt/lustre opts:
Stopping /mnt/lustre-mds2 (opts:-f) on oleg322-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg322-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg322-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg322-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg322-server'
oleg322-server: oleg322-server.virtnet: executing unload_modules_local
oleg322-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg322-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.
=== sanity-lfsck: start setup 20:00:24 (1767315624) ===
Stopping clients: oleg322-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg322-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 2
oleg322-server: oleg322-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg322-server'
oleg322-server: oleg322-server.virtnet: executing load_modules_local
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: Loading modules from /home/green/git/lustre-release/lustre
oleg322-server: detected 4 online CPUs by sysfs
oleg322-server: MODOPTS_LIBCFS=
oleg322-server: Force libcfs to create 2 CPU partitions
oleg322-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg322-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format ost1: /dev/vde
Format mds2: /dev/vdd
Format mds1: /dev/vdc
Format ost2: /dev/vdf
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Checking servers environments
Checking clients oleg322-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg322-server'
oleg322-server: oleg322-server.virtnet: executing load_modules_local
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: Loading modules from /home/green/git/lustre-release/lustre
oleg322-server: detected 4 online CPUs by sysfs
oleg322-server: MODOPTS_LIBCFS=
oleg322-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Commit the device label on /dev/vdc
Commit the device label on /dev/vdd
Started lustre-MDT0000
Started lustre-MDT0001
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0001-super.width=65536
seq.cli-lustre-OST0000-super.width=65536
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Commit the device label on /dev/vdf
Commit the device label on /dev/vde
Started lustre-OST0001
Started lustre-OST0000
/mnt/lustre-ost1: 0 B (0 bytes) trimmed
/mnt/lustre-ost2: 0 B (0 bytes) trimmed
Starting client: oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Starting client oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Started clients oleg322-client.virtnet: 
192.168.203.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9a549890e800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9a549890e800.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
Updated after 7s: want 'procname_uid' got 'procname_uid'
disable quota as required
osd-ldiskfs.track_declares_assert=1
=== sanity-lfsck: finish setup 20:06:14 (1767315974) ===

== sanity-lfsck test 0: Control LFSCK manually =========== 20:06:18 (1767315978)
preparing... 3 * 3 files will be created Thu Jan  1 20:06:20 EST 2026.
total: 3 mkdir in 0.03 seconds: 99.26 ops/second
total: 3 create in 0.03 seconds: 117.45 ops/second
total: 3 mkdir in 0.01 seconds: 281.27 ops/second
prepared Thu Jan  1 20:06:25 EST 2026.
fail_val=3
fail_loc=0x1600
Started LFSCK on the device lustre-MDT0000: scrub namespace
name: lfsck_namespace
magic: 0xa06249ff
version: 2
status: scanning-phase1
flags:
param:
last_completed_time: N/A
time_since_last_completed: N/A
latest_start_time: 1767315987
time_since_latest_start: 1 seconds
last_checkpoint_time: N/A
time_since_last_checkpoint: N/A
latest_start_position: 13, N/A, N/A
last_checkpoint_position: N/A, N/A, N/A
first_failure_position: N/A, N/A, N/A
checked_phase1: 0
checked_phase2: 0
updated_phase1: 0
updated_phase2: 0
failed_phase1: 0
failed_phase2: 0
directories: 0
dirent_repaired: 0
linkea_repaired: 0
nlinks_repaired: 0
multiple_linked_checked: 0
multiple_linked_repaired: 0
unknown_inconsistency: 0
unmatched_pairs_repaired: 0
dangling_repaired: 0
multiple_referenced_repaired: 0
bad_file_type_repaired: 0
lost_dirent_repaired: 0
local_lost_found_scanned: 0
local_lost_found_moved: 0
local_lost_found_skipped: 0
local_lost_found_failed: 0
striped_dirs_scanned: 0
striped_dirs_repaired: 0
striped_dirs_failed: 0
striped_dirs_disabled: 0
striped_dirs_skipped: 0
striped_shards_scanned: 0
striped_shards_repaired: 0
striped_shards_failed: 0
striped_shards_skipped: 0
name_hash_repaired: 0
linkea_overflow_cleared: 0
agent_entries_repaired: 0
success_count: 0
run_time_phase1: 1 seconds
run_time_phase2: 0 seconds
average_speed_phase1: 0 items/sec
average_speed_phase2: N/A
average_speed_total: 0 items/sec
real_time_speed_phase1: 0 items/sec
real_time_speed_phase2: N/A
current_position: 12, N/A, N/A
Stopped LFSCK on the device lustre-MDT0000.
Started LFSCK on the device lustre-MDT0000: scrub namespace
fail_loc=0
fail_val=0
Started LFSCK on the device lustre-MDT0000: scrub namespace
stopall, should NOT crash LU-3649
Stopping clients: oleg322-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg322-client.virtnet /mnt/lustre2 (opts:)
Stopping client oleg322-client.virtnet /mnt/lustre opts:
Stopping /mnt/lustre-mds1 (opts:-f) on oleg322-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg322-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg322-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg322-server
PASS 0 (49s)

== sanity-lfsck test 1a: LFSCK can find out and repair crashed FID-in-dirent ========================================================== 20:07:07 (1767316027)
Checking servers environments
Checking clients oleg322-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg322-server'
oleg322-server: oleg322-server.virtnet: executing load_modules_local
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: Loading modules from /home/green/git/lustre-release/lustre
oleg322-server: detected 4 online CPUs by sysfs
oleg322-server: MODOPTS_LIBCFS=
oleg322-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Started lustre-MDT0000
Started lustre-MDT0001
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0001-super.width=65536
seq.cli-lustre-OST0000-super.width=65536
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Started lustre-OST0001
Started lustre-OST0000
/mnt/lustre-ost2: 0 B (0 bytes) trimmed
/mnt/lustre-ost1: 0 B (0 bytes) trimmed
Starting client: oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Starting client oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Started clients oleg322-client.virtnet: 
192.168.203.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9a54873bf800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9a54873bf800.idle_timeout=debug
disable quota as required
preparing... 1 * 1 files will be created Thu Jan  1 20:10:12 EST 2026.
total: 1 mkdir in 0.01 seconds: 130.18 ops/second
total: 1 create in 0.01 seconds: 142.31 ops/second
total: 1 mkdir in 0.01 seconds: 113.55 ops/second
prepared Thu Jan  1 20:10:17 EST 2026.
fail_loc=0x1501
fail_loc=0
192.168.203.122@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg322-client.virtnet /mnt/lustre (opts:)
Started LFSCK on the device lustre-MDT0000: scrub namespace
Fix? no
Fix? no
Inode 126 ref count is 4, should be 3.  Fix? no
Inode 129 ref count is 3, should be 2.  Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
e2fsck -d -v -t -t -f -n /dev/vdc -m8
oleg322-server: e2fsck 1.47.3-wc2 (11-Nov-2025)
oleg322-server: Use max possible thread num: 1 instead
Warning!  /dev/vdc is in use.
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 1], used inodes 277/40000
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2316: increase inode 7 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 82 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 82 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 83 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 89 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 93 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 95 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 95 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 96 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 97 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 98 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 99 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 100 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 100 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 101 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 101 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 102 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 102 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 103 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 103 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 104 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 104 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 105 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 105 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 106 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 106 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 107 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 107 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 108 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 108 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 109 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 109 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 110 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 110 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 111 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 111 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 112 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 112 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 113 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 113 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 114 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 114 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 115 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 115 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 116 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 116 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 117 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 117 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 118 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 118 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 119 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 119 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 120 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 120 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 121 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 121 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 122 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 123 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 123 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 124 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 126 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2316: increase inode 127 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 127 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 128 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 128 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 129 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 129 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 130 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 130 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 134 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 134 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 135 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 135 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 136 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 137 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 137 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 138 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 138 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 139 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 139 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 140 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 140 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 141 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 141 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 142 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 142 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 143 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 143 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 144 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 144 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 145 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 145 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 146 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 146 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 147 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 147 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 148 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 148 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 149 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 149 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 150 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 150 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 151 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 151 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 152 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 152 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 153 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 153 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 154 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 155 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 155 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 156 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 156 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 157 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 157 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 158 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 158 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 159 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 159 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 160 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 160 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 161 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 161 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 162 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 162 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 163 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 163 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 164 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 164 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 165 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 165 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 166 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 166 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 167 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 167 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 168 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 168 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 169 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 169 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 170 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 170 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 174 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 175 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 176 badness 0 to 2 for 10084
[Thread 0] group 0 finished
[Thread 0] e2fsck_pass1_run:2316: increase inode 20036 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 20036 badness 1 to 3 for 10084
[Thread 0] group 1 finished
[Thread 0] Pass 1: Memory used: 264k/0k (147k/118k), time:  0.02/ 0.00/ 0.00
[Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 42.35MB/s
[Thread 0] Scanned group range [0, 1], used inodes 277/277
Pass 2: Checking directory structure
check_dir_block:1649: increase inode 126 badness 1 to 2 for 20047
Entry 'd1a.sanity-lfsck' in /ROOT (126) references inode 20098 fid=[0x200000bd1:0x1:0x0] found in group 1's unused inodes area.
Fix? no

Entry 'd1a.sanity-lfsck' in /ROOT (126) has deleted/unused inode 20098 fid=[0xd10b000002000000:0x1oleg322-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (1650688, 265) != expected (1667072, 350)
oleg322-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (1650688, 265) != expected (1667072, 350)
oleg322-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (1650688, 265) != expected (1667072, 350)
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 4
000000:0x0].  Clear? no

check_dir_block:1649: increase inode 129 badness 3 to 4 for 20047
Entry 'MDT0000' in /ROOT/.lustre/lost+found (129) references inode 20105 fid=[0x200000403:0x1:0x0] found in group 1's unused inodes area.
Fix? no

Entry 'MDT0000' in /ROOT/.lustre/lost+found (129) has deleted/unused inode 20105 fid=[0x304000002000000:0x1000000:0x0].  Clear? no

Pass 2: Memory used: 264k/0k (100k/165k), time:  0.01/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 71.59MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 264k/0k (100k/165k), time:  0.06/ 0.00/ 0.00
Pass 3: Memory used: 264k/0k (98k/167k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 4: Checking reference counts
Inode 126 ref count is 4, should be 3.  Fix? no

Inode 129 ref count is 3, should be 2.  Fix? no

Pass 4: Memory used: 264k/0k (82k/182k), time:  0.00/ 0.00/ 0.00
Pass 4: I/O read: 1MB, write: 0MB, rate: 285.14MB/s
Pass 5: Checking group summary information
Block bitmap differences:  -12159 -(19840--19846)
Fix? no

Free blocks count wrong for group #0 (7745, counted=7737).
Fix? no

Free blocks count wrong for group #1 (4753, counted=4769).
Fix? no

Free blocks count wrong (12908, counted=12506).
Fix? no

Inode bitmap differences:  -(181--270) -(20098--20105)
Fix? no

Free inodes count wrong for group #0 (19820, counted=19732).
Fix? no

Free inodes count wrong for group #1 (19903, counted=19895).
Fix? no

Free inodes count wrong (39986, counted=39627).
Fix? no

Pass 5: Memory used: 376k/0k (82k/295k), time:  0.01/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 75.00MB/s
Update quota info for quota type 0? no

Update quota info for quota type 1? no

Update quota info for quota type 2? no


lustre-MDT0000: ********** WARNING: Filesystem still has errors **********


          14 inodes used (0.04%, out of 40000)
           2 non-contiguous files (14.3%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 203
       12092 blocks used (48.37%, out of 25000)
           0 bad blocks
           1 large file

         148 regular files
         117 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         267 files
Memory used: 376k/0k (84k/293k), time:  0.09/ 0.00/ 0.00
I/O read: 1MB, write: 0MB, rate: 11.68MB/s
 sanity-lfsck test_1a: @@@@@@ FAIL: (2) Detected inconsistency on mds1 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7487:error()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:129:run_e2fsck_on_mds_facet()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:262:test_1a()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7863:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7741:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:272:main()
Dumping lctl log to /tmp/testlogs//sanity-lfsck.test_1a.*.1767316237.log
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_1a.debug_log.oleg322-server.1767316237.log.5JUmY4" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_1a.dmesg.oleg322-server.1767316237.log.DvrpwQ" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]
FAIL 1a (215s)

== sanity-lfsck test 1b: LFSCK can find out and repair the missing FID-in-LMA ========================================================== 20:10:42 (1767316242)
Checking servers environments
Checking clients oleg322-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg322-server'
oleg322-server: oleg322-server.virtnet: executing load_modules_local
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: Loading modules from /home/green/git/lustre-release/lustre
oleg322-server: detected 4 online CPUs by sysfs
oleg322-server: MODOPTS_LIBCFS=
oleg322-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Starting client oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Started clients oleg322-client.virtnet: 
192.168.203.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9a549247a000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9a549247a000.idle_timeout=debug
disable quota as required
preparing... 1 * 1 files will be created Thu Jan  1 20:13:32 EST 2026.
total: 1 mkdir in 0.01 seconds: 180.18 ops/second
total: 1 create in 0.01 seconds: 198.17 ops/second
total: 1 mkdir in 0.01 seconds: 158.56 ops/second
prepared Thu Jan  1 20:13:37 EST 2026.
fail_loc=0x1502
fail_loc=0
192.168.203.122@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg322-client.virtnet /mnt/lustre (opts:)
fail_loc=0x1506
Started LFSCK on the device lustre-MDT0000: scrub namespace
fail_loc=0
Fix? no
Fix? no
Inode 126 ref count is 4, should be 3.  Fix? no
Inode 129 ref count is 3, should be 2.  Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
e2fsck -d -v -t -t -f -n /dev/vdc -m8
oleg322-server: e2fsck 1.47.3-wc2 (11-Nov-2025)
oleg322-server: Use max possible thread num: 1 instead
Warning!  /dev/vdc is in use.
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 1], used inodes 277/40000
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2316: increase inode 7 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 82 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 82 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 83 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 89 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 93 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 95 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 95 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 96 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 97 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 98 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 99 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 100 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 100 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 101 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 101 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 102 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 102 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 103 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 103 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 104 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 104 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 105 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 105 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 106 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 106 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 107 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 107 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 108 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 108 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 109 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 109 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 110 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 110 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 111 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 111 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 112 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 112 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 113 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 113 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 114 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 114 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 115 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 115 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 116 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 116 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 117 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 117 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 118 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 118 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 119 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 119 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 120 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 120 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 121 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 121 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 122 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 123 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 123 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 124 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 126 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2316: increase inode 127 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 127 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 128 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 128 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 129 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 129 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 130 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 130 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 134 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 134 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 135 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 135 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 136 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 137 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 137 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 138 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 138 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 139 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 139 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 140 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 140 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 141 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 141 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 142 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 142 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 143 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 143 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 144 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 144 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 145 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 145 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 146 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 146 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 147 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 147 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 148 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 148 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 149 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 149 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 150 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 150 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 151 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 151 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 152 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 152 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 153 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 153 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 154 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 155 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 155 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 156 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 156 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 157 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 157 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 158 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 158 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 159 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 159 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 160 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 160 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 161 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 161 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 162 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 162 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 163 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 163 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 164 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 164 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 165 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 165 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 166 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 166 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 167 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 167 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 168 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 168 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 169 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 169 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 170 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 170 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 174 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 175 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 176 badness 0 to 2 for 10084
[Thread 0] group 0 finished
[Thread 0] e2fsck_pass1_run:2316: increase inode 20036 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 20036 badness 1 to 3 for 10084
[Thread 0] group 1 finished
[Thread 0] Pass 1: Memory used: 264k/0k (147k/118k), time:  0.01/ 0.00/ 0.00
[Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 143.84MB/s
[Thread 0] Scanned group range [0, 1], used inodes 277/277
Pass 2: Checking directory structure
check_dir_block:1649: increase inode 126 badness 1 to 2 for 20047
Entry 'd1a.sanity-lfsck' in /ROOT (126) references inode 20098 fid=[0x200000bd1:0x1:0x0] found in group 1's unused inodes area.
Fix? no

Entry 'd1a.sanity-lfsck' in /ROOT (126) has deleted/unused inode 20098 fid=[0xd10b000002000000:0xoleg322-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (1650688, 265) != expected (1667072, 350)
oleg322-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (1650688, 265) != expected (1667072, 350)
oleg322-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (1650688, 265) != expected (1667072, 350)
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 4
1000000:0x0].  Clear? no

check_dir_block:1649: increase inode 129 badness 3 to 4 for 20047
Entry 'MDT0000' in /ROOT/.lustre/lost+found (129) references inode 20105 fid=[0x200000403:0x1:0x0] found in group 1's unused inodes area.
Fix? no

Entry 'MDT0000' in /ROOT/.lustre/lost+found (129) has deleted/unused inode 20105 fid=[0x304000002000000:0x1000000:0x0].  Clear? no

Pass 2: Memory used: 264k/0k (100k/165k), time:  0.01/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 124.81MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 264k/0k (100k/165k), time:  0.03/ 0.00/ 0.00
Pass 3: Memory used: 264k/0k (98k/167k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 4: Checking reference counts
Inode 126 ref count is 4, should be 3.  Fix? no

Inode 129 ref count is 3, should be 2.  Fix? no

Pass 4: Memory used: 264k/0k (82k/182k), time:  0.00/ 0.00/ 0.00
Pass 4: I/O read: 1MB, write: 0MB, rate: 399.52MB/s
Pass 5: Checking group summary information
Block bitmap differences:  -12159 -(19840--19846)
Fix? no

Free blocks count wrong for group #0 (7745, counted=7737).
Fix? no

Free blocks count wrong for group #1 (4753, counted=4769).
Fix? no

Free blocks count wrong (12908, counted=12506).
Fix? no

Inode bitmap differences:  -(181--270) -(20098--20105)
Fix? no

Free inodes count wrong for group #0 (19820, counted=19732).
Fix? no

Free inodes count wrong for group #1 (19903, counted=19895).
Fix? no

Free inodes count wrong (39986, counted=39627).
Fix? no

Pass 5: Memory used: 376k/0k (82k/295k), time:  0.02/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 59.08MB/s
Update quota info for quota type 0? no

Update quota info for quota type 1? no

Update quota info for quota type 2? no


lustre-MDT0000: ********** WARNING: Filesystem still has errors **********


          14 inodes used (0.04%, out of 40000)
           2 non-contiguous files (14.3%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 203
       12092 blocks used (48.37%, out of 25000)
           0 bad blocks
           1 large file

         148 regular files
         117 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         267 files
Memory used: 376k/0k (84k/293k), time:  0.05/ 0.01/ 0.00
I/O read: 1MB, write: 0MB, rate: 19.50MB/s
 sanity-lfsck test_1b: @@@@@@ FAIL: (2) Detected inconsistency on mds1 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7487:error()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:129:run_e2fsck_on_mds_facet()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:308:test_1b()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7863:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7741:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:318:main()
Dumping lctl log to /tmp/testlogs//sanity-lfsck.test_1b.*.1767316438.log
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_1b.debug_log.oleg322-server.1767316438.log.JOL1Mx" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_1b.dmesg.oleg322-server.1767316438.log.m8PWIf" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]
FAIL 1b (201s)

== sanity-lfsck test 1c: LFSCK can find out and repair lost FID-in-dirent ========================================================== 20:14:03 (1767316443)
Checking servers environments
Checking clients oleg322-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg322-server'
oleg322-server: oleg322-server.virtnet: executing load_modules_local
oleg322-server: Loading modules from /home/green/git/lustre-release/lustre
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: detected 4 online CPUs by sysfs
oleg322-server: MODOPTS_LIBCFS=
oleg322-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Starting client oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Started clients oleg322-client.virtnet: 
192.168.203.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9a5491ade800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9a5491ade800.idle_timeout=debug
disable quota as required
preparing... 1 * 1 files will be created Thu Jan  1 20:16:58 EST 2026.
total: 1 mkdir in 0.01 seconds: 114.90 ops/second
total: 1 create in 0.01 seconds: 146.01 ops/second
total: 1 mkdir in 0.01 seconds: 105.95 ops/second
prepared Thu Jan  1 20:17:02 EST 2026.
fail_loc=0x1504
fail_loc=0
192.168.203.122@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg322-client.virtnet /mnt/lustre (opts:)
Started LFSCK on the device lustre-MDT0000: scrub namespace
Fix? no
Fix? no
Inode 126 ref count is 4, should be 3.  Fix? no
Inode 129 ref count is 3, should be 2.  Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
e2fsck -d -v -t -t -f -n /dev/vdc -m8
oleg322-server: e2fsck 1.47.3-wc2 (11-Nov-2025)
oleg322-server: Use max possible thread num: 1 instead
oleg322-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (1650688, 265) != expected (1667072, 350)
Warning!  /dev/vdc is in use.
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 1], used inodes 277/40000
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2316: increase inode 7 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 82 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 82 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 83 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 89 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 93 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 95 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 95 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 96 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 97 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 98 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 99 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 100 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 100 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 101 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 101 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 102 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 102 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 103 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 103 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 104 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 104 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 105 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 105 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 106 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 106 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 107 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 107 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 108 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 108 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 109 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 109 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 110 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 110 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 111 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 111 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 112 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 112 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 113 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 113 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 114 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 114 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 115 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 115 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 116 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 116 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 117 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 117 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 118 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 118 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 119 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 119 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 120 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 120 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 121 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 121 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 122 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 123 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 123 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 124 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 126 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2316: increase inode 127 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 127 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 128 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 128 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 129 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 129 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 130 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 130 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 134 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 134 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 135 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 135 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 136 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 137 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 137 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 138 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 138 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 139 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 139 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 140 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 140 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 141 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 141 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 142 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 142 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 143 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 143 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 144 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 144 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 145 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 145 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 146 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 146 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 147 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 147 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 148 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 148 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 149 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 149 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 150 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 150 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 151 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 151 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 152 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 152 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 153 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 153 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 154 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 155 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 155 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 156 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 156 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 157 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 157 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 158 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 158 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 159 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 159 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 160 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 160 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 161 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 161 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 162 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 162 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 163 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 163 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 164 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 164 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 165 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 165 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 166 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 166 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 167 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 167 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 168 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 168 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 169 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 169 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 170 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 170 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 174 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 175 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 176 badness 0 to 2 for 10084
[Thread 0] group 0 finished
[Thread 0] e2fsck_pass1_run:2316: increase inode 20036 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 20036 badness 1 to 3 for 10084
[Thread 0] group 1 finished
[Thread 0] Pass 1: Memory used: 264k/0k (147k/118k), time:  0.00/ 0.00/ 0.00
[Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 461.89MB/s
[Thread 0] Scanned group range [0, 1], used inodes 277/277
Pass 2: Checking directory structure
check_dir_block:1649: increase inode 126 badness 1 to 2 for 20047
Entry 'd1a.sanity-lfsck' in /ROOT (126) references inode 20098 fid=[0x200000bd1:0x1:0x0] found in group 1's unused inodes area.
Fix? no

Entry 'd1a.sanity-lfsck' in /ROOT (126) has deleted/unused inode 20098 fid=[0xd10b000002000000:0xoleg322-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (1650688, 265) != expected (1667072, 350)
oleg322-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (1650688, 265) != expected (1667072, 350)
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 4
1000000:0x0].  Clear? no

check_dir_block:1649: increase inode 129 badness 3 to 4 for 20047
Entry 'MDT0000' in /ROOT/.lustre/lost+found (129) references inode 20105 fid=[0x200000403:0x1:0x0] found in group 1's unused inodes area.
Fix? no

Entry 'MDT0000' in /ROOT/.lustre/lost+found (129) has deleted/unused inode 20105 fid=[0x304000002000000:0x1000000:0x0].  Clear? no

Pass 2: Memory used: 264k/0k (100k/165k), time:  0.00/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 439.56MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 264k/0k (100k/165k), time:  0.01/ 0.00/ 0.00
Pass 3: Memory used: 264k/0k (98k/167k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 4: Checking reference counts
Inode 126 ref count is 4, should be 3.  Fix? no

Inode 129 ref count is 3, should be 2.  Fix? no

Pass 4: Memory used: 264k/0k (82k/182k), time:  0.00/ 0.00/ 0.00
Pass 4: I/O read: 1MB, write: 0MB, rate: 277.70MB/s
Pass 5: Checking group summary information
Block bitmap differences:  -12159 -(19840--19846)
Fix? no

Free blocks count wrong for group #0 (7745, counted=7737).
Fix? no

Free blocks count wrong for group #1 (4753, counted=4769).
Fix? no

Free blocks count wrong (12908, counted=12506).
Fix? no

Inode bitmap differences:  -(181--270) -(20098--20105)
Fix? no

Free inodes count wrong for group #0 (19820, counted=19732).
Fix? no

Free inodes count wrong for group #1 (19903, counted=19895).
Fix? no

Free inodes count wrong (39986, counted=39627).
Fix? no

Pass 5: Memory used: 376k/0k (82k/295k), time:  0.01/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 181.95MB/s
Update quota info for quota type 0? no

Update quota info for quota type 1? no

Update quota info for quota type 2? no


lustre-MDT0000: ********** WARNING: Filesystem still has errors **********


          14 inodes used (0.04%, out of 40000)
           2 non-contiguous files (14.3%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 203
       12092 blocks used (48.37%, out of 25000)
           0 bad blocks
           1 large file

         148 regular files
         117 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         267 files
Memory used: 376k/0k (84k/293k), time:  0.02/ 0.01/ 0.00
I/O read: 1MB, write: 0MB, rate: 43.13MB/s
 sanity-lfsck test_1c: @@@@@@ FAIL: (2) Detected inconsistency on mds1 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7487:error()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:129:run_e2fsck_on_mds_facet()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:347:test_1c()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7863:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7741:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:357:main()
Dumping lctl log to /tmp/testlogs//sanity-lfsck.test_1c.*.1767316644.log
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_1c.debug_log.oleg322-server.1767316644.log.pIuv1b" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_1c.dmesg.oleg322-server.1767316644.log.qKIPwH" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]
FAIL 1c (206s)

== sanity-lfsck test 2a: LFSCK can find out and repair crashed linkEA entry ========================================================== 20:17:30 (1767316650)
Checking servers environments
Checking clients oleg322-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg322-server'
oleg322-server: oleg322-server.virtnet: executing load_modules_local
oleg322-server: Loading modules from /home/green/git/lustre-release/lustre
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: detected 4 online CPUs by sysfs
oleg322-server: MODOPTS_LIBCFS=
oleg322-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Starting client: oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Starting client oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Started clients oleg322-client.virtnet: 
192.168.203.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9a54873be800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9a54873be800.idle_timeout=debug
disable quota as required
preparing... 1 * 1 files will be created Thu Jan  1 20:20:22 EST 2026.
total: 1 mkdir in 0.01 seconds: 144.21 ops/second
total: 1 create in 0.01 seconds: 115.66 ops/second
total: 1 mkdir in 0.01 seconds: 137.37 ops/second
prepared Thu Jan  1 20:20:27 EST 2026.
fail_loc=0x1603
fail_loc=0
192.168.203.122@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg322-client.virtnet /mnt/lustre (opts:)
Started LFSCK on the device lustre-MDT0000: scrub namespace
Fix? no
Fix? no
Inode 126 ref count is 4, should be 3.  Fix? no
Inode 129 ref count is 3, should be 2.  Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
e2fsck -d -v -t -t -f -n /dev/vdc -m8
oleg322-server: e2fsck 1.47.3-wc2 (11-Nov-2025)
oleg322-server: Use max possible thread num: 1 instead
Warning!  /dev/vdc is in use.
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 1], used inodes 277/40000
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2316: increase inode 7 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 82 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 82 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 83 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 89 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 93 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 95 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 95 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 96 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 97 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 98 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 99 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 100 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 100 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 101 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 101 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 102 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 102 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 103 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 103 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 104 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 104 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 105 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 105 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 106 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 106 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 107 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 107 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 108 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 108 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 109 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 109 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 110 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 110 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 111 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 111 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 112 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 112 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 113 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 113 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 114 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 114 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 115 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 115 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 116 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 116 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 117 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 117 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 118 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 118 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 119 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 119 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 120 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 120 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 121 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 121 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 122 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 123 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 123 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 124 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 126 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2316: increase inode 127 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 127 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 128 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 128 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 129 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 129 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 130 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 130 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 134 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 134 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 135 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 135 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 136 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 137 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 137 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 138 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 138 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 139 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 139 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 140 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 140 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 141 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 141 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 142 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 142 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 143 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 143 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 144 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 144 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 145 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 145 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 146 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 146 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 147 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 147 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 148 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 148 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 149 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 149 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 150 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 150 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 151 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 151 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 152 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 152 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 153 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 153 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 154 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 155 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 155 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 156 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 156 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 157 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 157 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 158 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 158 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 159 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 159 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 160 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 160 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 161 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 161 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 162 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 162 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 163 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 163 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 164 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 164 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 165 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 165 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 166 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 166 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 167 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 167 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 168 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 168 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 169 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 169 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 170 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 170 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 174 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 175 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 176 badness 0 to 2 for 10084
[Thread 0] group 0 finished
[Thread 0] e2fsck_pass1_run:2316: increase inode 20036 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 20036 badness 1 to 3 for 10084
[Thread 0] group 1 finished
[Thread 0] Pass 1: Memory used: 264k/0k (147k/118k), time:  0.02/ 0.00/ 0.00
[Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 55.90MB/s
[Thread 0] Scanned group range [0, 1], used inodes 277/277
Pass 2: Checking directory structure
check_dir_block:1649: increase inode 126 badness 1 to 2 for 20047
Entry 'd1a.sanity-lfsck' in /ROOT (126) references inode 20098 fid=[0x200000bd1:0x1:0x0] found in group 1's unused inodes area.
Fix? no

Entry 'd1a.sanity-lfsck' in /ROOT (126) has deleted/unused inode 20098 fid=[0xd10b000002000000:0x1oleg322-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (1650688, 265) != expected (1667072, 350)
oleg322-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (1650688, 265) != expected (1667072, 350)
oleg322-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (1650688, 265) != expected (1667072, 350)
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 4
000000:0x0].  Clear? no

check_dir_block:1649: increase inode 129 badness 3 to 4 for 20047
Entry 'MDT0000' in /ROOT/.lustre/lost+found (129) references inode 20105 fid=[0x200000403:0x1:0x0] found in group 1's unused inodes area.
Fix? no

Entry 'MDT0000' in /ROOT/.lustre/lost+found (129) has deleted/unused inode 20105 fid=[0x304000002000000:0x1000000:0x0].  Clear? no

Pass 2: Memory used: 264k/0k (100k/165k), time:  0.00/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 610.87MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 264k/0k (100k/165k), time:  0.05/ 0.00/ 0.00
Pass 3: Memory used: 264k/0k (98k/167k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 4: Checking reference counts
Inode 126 ref count is 4, should be 3.  Fix? no

Inode 129 ref count is 3, should be 2.  Fix? no

Pass 4: Memory used: 264k/0k (82k/182k), time:  0.01/ 0.00/ 0.00
Pass 4: I/O read: 1MB, write: 0MB, rate: 195.39MB/s
Pass 5: Checking group summary information
Block bitmap differences:  -12159 -(19840--19846)
Fix? no

Free blocks count wrong for group #0 (7745, counted=7737).
Fix? no

Free blocks count wrong for group #1 (4753, counted=4769).
Fix? no

Free blocks count wrong (12908, counted=12506).
Fix? no

Inode bitmap differences:  -(181--270) -(20098--20105)
Fix? no

Free inodes count wrong for group #0 (19820, counted=19732).
Fix? no

Free inodes count wrong for group #1 (19903, counted=19895).
Fix? no

Free inodes count wrong (39986, counted=39627).
Fix? no

Pass 5: Memory used: 376k/0k (82k/295k), time:  0.01/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 149.01MB/s
Update quota info for quota type 0? no

Update quota info for quota type 1? no

Update quota info for quota type 2? no


lustre-MDT0000: ********** WARNING: Filesystem still has errors **********


          14 inodes used (0.04%, out of 40000)
           2 non-contiguous files (14.3%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 203
       12092 blocks used (48.37%, out of 25000)
           0 bad blocks
           1 large file

         148 regular files
         117 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         267 files
Memory used: 376k/0k (84k/293k), time:  0.06/ 0.01/ 0.00
I/O read: 1MB, write: 0MB, rate: 17.04MB/s
 sanity-lfsck test_2a: @@@@@@ FAIL: (2) Detected inconsistency on mds1 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7487:error()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:129:run_e2fsck_on_mds_facet()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:386:test_2a()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7863:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7741:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:398:main()
Dumping lctl log to /tmp/testlogs//sanity-lfsck.test_2a.*.1767316847.log
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_2a.debug_log.oleg322-server.1767316847.log.l4xJpT" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_2a.dmesg.oleg322-server.1767316847.log.zUznPk" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]
FAIL 2a (202s)

== sanity-lfsck test 2b: LFSCK can find out and remove invalid linkEA entry ========================================================== 20:20:52 (1767316852)
Checking servers environments
Checking clients oleg322-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg322-server'
oleg322-server: oleg322-server.virtnet: executing load_modules_local
oleg322-server: Loading modules from /home/green/git/lustre-release/lustre
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: detected 4 online CPUs by sysfs
oleg322-server: MODOPTS_LIBCFS=
oleg322-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Starting client oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Started clients oleg322-client.virtnet: 
192.168.203.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9a54873bd000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9a54873bd000.idle_timeout=debug
disable quota as required
preparing... 1 * 1 files will be created Thu Jan  1 20:23:18 EST 2026.
total: 1 mkdir in 0.01 seconds: 97.30 ops/second
total: 1 create in 0.01 seconds: 176.49 ops/second
total: 1 mkdir in 0.01 seconds: 78.47 ops/second
prepared Thu Jan  1 20:23:22 EST 2026.
fail_loc=0x1604
fail_loc=0
192.168.203.122@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg322-client.virtnet /mnt/lustre (opts:)
Started LFSCK on the device lustre-MDT0000: scrub namespace
Fix? no
Fix? no
Inode 126 ref count is 4, should be 3.  Fix? no
Inode 129 ref count is 3, should be 2.  Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
e2fsck -d -v -t -t -f -n /dev/vdc -m8
oleg322-server: e2fsck 1.47.3-wc2 (11-Nov-2025)
oleg322-server: Use max possible thread num: 1 instead
Warning!  /dev/vdc is in use.
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 1], used inodes 277/40000
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2316: increase inode 7 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 82 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 82 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 83 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 89 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 93 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 95 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 95 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 96 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 97 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 98 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 99 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 100 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 100 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 101 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 101 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 102 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 102 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 103 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 103 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 104 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 104 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 105 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 105 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 106 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 106 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 107 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 107 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 108 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 108 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 109 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 109 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 110 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 110 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 111 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 111 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 112 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 112 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 113 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 113 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 114 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 114 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 115 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 115 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 116 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 116 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 117 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 117 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 118 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 118 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 119 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 119 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 120 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 120 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 121 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 121 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 122 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 123 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 123 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 124 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 126 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2316: increase inode 127 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 127 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 128 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 128 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 129 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 129 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 130 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 130 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 134 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 134 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 135 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 135 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 136 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 137 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 137 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 138 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 138 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 139 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 139 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 140 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 140 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 141 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 141 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 142 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 142 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 143 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 143 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 144 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 144 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 145 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 145 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 146 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 146 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 147 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 147 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 148 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 148 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 149 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 149 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 150 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 150 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 151 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 151 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 152 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 152 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 153 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 153 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 154 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 155 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 155 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 156 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 156 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 157 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 157 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 158 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 158 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 159 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 159 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 160 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 160 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 161 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 161 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 162 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 162 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 163 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 163 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 164 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 164 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 165 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 165 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 166 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 166 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 167 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 167 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 168 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 168 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 169 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 169 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 170 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 170 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 174 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 175 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 176 badness 0 to 2 for 10084
[Thread 0] group 0 finished
[Thread 0] e2fsck_pass1_run:2316: increase inode 20036 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 20036 badness 1 to 3 for 10084
[Thread 0] group 1 finished
[Thread 0] Pass 1: Memory used: 264k/0k (147k/118k), time:  0.01/ 0.00/ 0.00
[Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 89.79MB/s
[Thread 0] Scanned group range [0, 1], used inodes 277/277
Pass 2: Checking directory structure
check_dir_block:1649: increase inode 126 badness 1 to 2 for 20047
Entry 'd1a.sanity-lfsck' in /ROOT (126) references inode 20098 fid=[0x200000bd1:0x1:0x0] found in group 1's unused inodes area.
Fix? no

Entry 'd1a.sanity-lfsck' in /ROOT (126) has deleted/unused inode 20098 fid=[0xd10b000002000000:0x1oleg322-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (1650688, 265) != expected (1667072, 350)
oleg322-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (1650688, 265) != expected (1667072, 350)
oleg322-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (1650688, 265) != expected (1667072, 350)
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 4
000000:0x0].  Clear? no

check_dir_block:1649: increase inode 129 badness 3 to 4 for 20047
Entry 'MDT0000' in /ROOT/.lustre/lost+found (129) references inode 20105 fid=[0x200000403:0x1:0x0] found in group 1's unused inodes area.
Fix? no

Entry 'MDT0000' in /ROOT/.lustre/lost+found (129) has deleted/unused inode 20105 fid=[0x304000002000000:0x1000000:0x0].  Clear? no

Pass 2: Memory used: 264k/0k (100k/165k), time:  0.00/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 622.67MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 264k/0k (100k/165k), time:  0.05/ 0.00/ 0.00
Pass 3: Memory used: 264k/0k (98k/167k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 4: Checking reference counts
Inode 126 ref count is 4, should be 3.  Fix? no

Inode 129 ref count is 3, should be 2.  Fix? no

Pass 4: Memory used: 264k/0k (82k/182k), time:  0.00/ 0.00/ 0.00
Pass 4: I/O read: 1MB, write: 0MB, rate: 926.78MB/s
Pass 5: Checking group summary information
Block bitmap differences:  -12159 -(19840--19846)
Fix? no

Free blocks count wrong for group #0 (7745, counted=7737).
Fix? no

Free blocks count wrong for group #1 (4753, counted=4769).
Fix? no

Free blocks count wrong (12908, counted=12506).
Fix? no

Inode bitmap differences:  -(181--270) -(20098--20105)
Fix? no

Free inodes count wrong for group #0 (19820, counted=19732).
Fix? no

Free inodes count wrong for group #1 (19903, counted=19895).
Fix? no

Free inodes count wrong (39986, counted=39627).
Fix? no

Pass 5: Memory used: 376k/0k (82k/295k), time:  0.02/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 46.28MB/s
Update quota info for quota type 0? no

Update quota info for quota type 1? no

Update quota info for quota type 2? no


lustre-MDT0000: ********** WARNING: Filesystem still has errors **********


          14 inodes used (0.04%, out of 40000)
           2 non-contiguous files (14.3%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 203
       12092 blocks used (48.37%, out of 25000)
           0 bad blocks
           1 large file

         148 regular files
         117 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         267 files
Memory used: 376k/0k (84k/293k), time:  0.07/ 0.01/ 0.00
I/O read: 1MB, write: 0MB, rate: 13.83MB/s
 sanity-lfsck test_2b: @@@@@@ FAIL: (2) Detected inconsistency on mds1 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7487:error()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:129:run_e2fsck_on_mds_facet()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:423:test_2b()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7863:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7741:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:435:main()
Dumping lctl log to /tmp/testlogs//sanity-lfsck.test_2b.*.1767317025.log
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_2b.debug_log.oleg322-server.1767317025.log.i5dcPJ" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_2b.dmesg.oleg322-server.1767317025.log.Eu7RBb" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]
FAIL 2b (179s)

== sanity-lfsck test 2c: LFSCK can find out and remove repeated linkEA entry ========================================================== 20:23:51 (1767317031)
Checking servers environments
Checking clients oleg322-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg322-server'
oleg322-server: oleg322-server.virtnet: executing load_modules_local
oleg322-server: Loading modules from /home/green/git/lustre-release/lustre
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: detected 4 online CPUs by sysfs
oleg322-server: MODOPTS_LIBCFS=
oleg322-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Starting client oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Started clients oleg322-client.virtnet: 
192.168.203.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9a549890e000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9a549890e000.idle_timeout=debug
disable quota as required
preparing... 1 * 1 files will be created Thu Jan  1 20:26:35 EST 2026.
total: 1 mkdir in 0.01 seconds: 98.09 ops/second
total: 1 create in 0.01 seconds: 92.87 ops/second
total: 1 mkdir in 0.01 seconds: 120.61 ops/second
prepared Thu Jan  1 20:26:41 EST 2026.
fail_loc=0x1605
fail_loc=0
192.168.203.122@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg322-client.virtnet /mnt/lustre (opts:)
Started LFSCK on the device lustre-MDT0000: scrub namespace
Fix? no
Fix? no
Inode 126 ref count is 4, should be 3.  Fix? no
Inode 129 ref count is 3, should be 2.  Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
e2fsck -d -v -t -t -f -n /dev/vdc -m8
oleg322-server: e2fsck 1.47.3-wc2 (11-Nov-2025)
oleg322-server: Use max possible thread num: 1 instead
Warning!  /dev/vdc is in use.
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 1], used inodes 277/40000
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2316: increase inode 7 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 82 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 82 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 83 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 89 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 93 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 95 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 95 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 96 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 97 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 98 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 99 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 100 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 100 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 101 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 101 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 102 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 102 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 103 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 103 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 104 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 104 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 105 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 105 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 106 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 106 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 107 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 107 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 108 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 108 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 109 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 109 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 110 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 110 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 111 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 111 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 112 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 112 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 113 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 113 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 114 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 114 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 115 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 115 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 116 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 116 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 117 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 117 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 118 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 118 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 119 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 119 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 120 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 120 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 121 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 121 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 122 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 123 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 123 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 124 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 126 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2316: increase inode 127 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 127 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 128 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 128 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 129 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 129 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 130 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 130 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 134 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 134 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 135 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 135 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 136 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 137 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 137 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 138 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 138 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 139 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 139 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 140 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 140 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 141 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 141 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 142 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 142 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 143 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 143 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 144 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 144 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 145 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 145 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 146 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 146 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 147 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 147 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 148 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 148 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 149 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 149 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 150 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 150 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 151 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 151 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 152 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 152 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 153 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 153 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 154 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 155 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 155 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 156 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 156 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 157 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 157 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 158 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 158 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 159 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 159 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 160 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 160 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 161 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 161 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 162 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 162 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 163 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 163 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 164 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 164 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 165 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 165 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 166 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 166 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 167 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 167 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 168 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 168 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 169 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 169 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 170 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 170 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 174 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 175 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 176 badness 0 to 2 for 10084
[Thread 0] group 0 finished
[Thread 0] e2fsck_pass1_run:2316: increase inode 20036 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 20036 badness 1 to 3 for 10084
[Thread 0] group 1 finished
[Thread 0] Pass 1: Memory used: 264k/0k (147k/118k), time:  0.01/ 0.00/ 0.00
[Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 86.04MB/s
[Thread 0] Scanned group range [0, 1], used inodes 277/277
Pass 2: Checking directory structure
check_dir_block:1649: increase inode 126 badness 1 to 2 for 20047
Entry 'd1a.sanity-lfsck' in /ROOT (126) references inode 20098 fid=[0x200000bd1:0x1:0x0] found in group 1's unused inodes area.
Fix? no

Entry 'd1a.sanity-lfsck' in /ROOT (126) has deleted/unused inode 20098 fid=[0xd10b000002000000:0x1oleg322-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (1650688, 265) != expected (1667072, 350)
oleg322-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (1650688, 265) != expected (1667072, 350)
oleg322-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (1650688, 265) != expected (1667072, 350)
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 4
000000:0x0].  Clear? no

check_dir_block:1649: increase inode 129 badness 3 to 4 for 20047
Entry 'MDT0000' in /ROOT/.lustre/lost+found (129) references inode 20105 fid=[0x200000403:0x1:0x0] found in group 1's unused inodes area.
Fix? no

Entry 'MDT0000' in /ROOT/.lustre/lost+found (129) has deleted/unused inode 20105 fid=[0x304000002000000:0x1000000:0x0].  Clear? no

Pass 2: Memory used: 264k/0k (100k/165k), time:  0.01/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 169.35MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 264k/0k (100k/165k), time:  0.04/ 0.00/ 0.00
Pass 3: Memory used: 264k/0k (98k/167k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 4: Checking reference counts
Inode 126 ref count is 4, should be 3.  Fix? no

Inode 129 ref count is 3, should be 2.  Fix? no

Pass 4: Memory used: 264k/0k (82k/182k), time:  0.00/ 0.00/ 0.00
Pass 4: I/O read: 1MB, write: 0MB, rate: 218.15MB/s
Pass 5: Checking group summary information
Block bitmap differences:  -12159 -(19840--19846)
Fix? no

Free blocks count wrong for group #0 (7745, counted=7737).
Fix? no

Free blocks count wrong for group #1 (4753, counted=4769).
Fix? no

Free blocks count wrong (12908, counted=12506).
Fix? no

Inode bitmap differences:  -(181--270) -(20098--20105)
Fix? no

Free inodes count wrong for group #0 (19820, counted=19732).
Fix? no

Free inodes count wrong for group #1 (19903, counted=19895).
Fix? no

Free inodes count wrong (39986, counted=39627).
Fix? no

Pass 5: Memory used: 376k/0k (82k/295k), time:  0.01/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 115.22MB/s
Update quota info for quota type 0? no

Update quota info for quota type 1? no

Update quota info for quota type 2? no


lustre-MDT0000: ********** WARNING: Filesystem still has errors **********


          14 inodes used (0.04%, out of 40000)
           2 non-contiguous files (14.3%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 203
       12092 blocks used (48.37%, out of 25000)
           0 bad blocks
           1 large file

         148 regular files
         117 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         267 files
Memory used: 376k/0k (84k/293k), time:  0.05/ 0.01/ 0.01
I/O read: 1MB, write: 0MB, rate: 19.53MB/s
 sanity-lfsck test_2c: @@@@@@ FAIL: (2) Detected inconsistency on mds1 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7487:error()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:129:run_e2fsck_on_mds_facet()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:463:test_2c()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7863:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7741:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:475:main()
Dumping lctl log to /tmp/testlogs//sanity-lfsck.test_2c.*.1767317223.log
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_2c.debug_log.oleg322-server.1767317223.log.nYdIjO" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_2c.dmesg.oleg322-server.1767317223.log.vIzFVu" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]
FAIL 2c (197s)

== sanity-lfsck test 2d: LFSCK can recover the missing linkEA entry ========================================================== 20:27:09 (1767317229)
Checking servers environments
Checking clients oleg322-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg322-server'
oleg322-server: oleg322-server.virtnet: executing load_modules_local
oleg322-server: Loading modules from /home/green/git/lustre-release/lustre
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: detected 4 online CPUs by sysfs
oleg322-server: MODOPTS_LIBCFS=
oleg322-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Starting client oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Started clients oleg322-client.virtnet: 
192.168.203.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9a5493393800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9a5493393800.idle_timeout=debug
disable quota as required
preparing... 1 * 1 files will be created Thu Jan  1 20:30:01 EST 2026.
total: 1 mkdir in 0.01 seconds: 151.15 ops/second
total: 1 create in 0.01 seconds: 195.85 ops/second
total: 1 mkdir in 0.02 seconds: 57.95 ops/second
prepared Thu Jan  1 20:30:07 EST 2026.
fail_loc=0x161d
fail_loc=0
192.168.203.122@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg322-client.virtnet /mnt/lustre (opts:)
Started LFSCK on the device lustre-MDT0000: scrub namespace
Fix? no
Fix? no
Inode 126 ref count is 4, should be 3.  Fix? no
Inode 129 ref count is 3, should be 2.  Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
Fix? no
e2fsck -d -v -t -t -f -n /dev/vdc -m8
oleg322-server: e2fsck 1.47.3-wc2 (11-Nov-2025)
oleg322-server: Use max possible thread num: 1 instead
Warning!  /dev/vdc is in use.
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 1], used inodes 277/40000
[Thread 0] jumping to group 0
[Thread 0] e2fsck_pass1_run:2316: increase inode 7 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 81 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 82 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 82 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 83 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 89 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 93 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 95 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 95 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 96 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 97 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 98 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 99 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 100 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 100 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 101 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 101 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 102 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 102 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 103 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 103 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 104 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 104 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 105 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 105 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 106 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 106 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 107 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 107 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 108 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 108 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 109 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 109 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 110 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 110 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 111 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 111 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 112 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 112 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 113 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 113 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 114 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 114 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 115 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 115 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 116 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 116 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 117 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 117 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 118 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 118 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 119 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 119 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 120 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 120 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 121 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 121 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 122 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 123 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 123 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 124 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 126 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2316: increase inode 127 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 127 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 128 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 128 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 129 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 129 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 130 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 130 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 134 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 134 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 135 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 135 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 136 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 137 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 137 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 138 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 138 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 139 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 139 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 140 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 140 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 141 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 141 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 142 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 142 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 143 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 143 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 144 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 144 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 145 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 145 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 146 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 146 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 147 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 147 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 148 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 148 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 149 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 149 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 150 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 150 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 151 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 151 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 152 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 152 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 153 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 153 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 154 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 155 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 155 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 156 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 156 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 157 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 157 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 158 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 158 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 159 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 159 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 160 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 160 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 161 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 161 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 162 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 162 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 163 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 163 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 164 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 164 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 165 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 165 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 166 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 166 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 167 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 167 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 168 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 168 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 169 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 169 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2316: increase inode 170 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 170 badness 1 to 3 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 174 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 175 badness 0 to 2 for 10084
[Thread 0] e2fsck_pass1_run:2680: increase inode 176 badness 0 to 2 for 10084
[Thread 0] group 0 finished
[Thread 0] e2fsck_pass1_run:2316: increase inode 20036 badness 0 to 1 for 1005c
[Thread 0] e2fsck_pass1_run:2680: increase inode 20036 badness 1 to 3 for 10084
[Thread 0] group 1 finished
[Thread 0] Pass 1: Memory used: 264k/0k (147k/118k), time:  0.01/ 0.00/ 0.00
[Thread 0] Pass 1: I/O read: 1MB, write: 0MB, rate: 96.97MB/s
[Thread 0] Scanned group range [0, 1], used inodes 277/277
Pass 2: Checking directory structure
check_dir_block:1649: increase inode 126 badness 1 to 2 for 20047
Entry 'd1a.sanity-lfsck' in /ROOT (126) references inode 20098 fid=[0x200000bd1:0x1:0x0] found in group 1's unused inodes area.
Fix? no

Entry 'd1a.sanity-lfsck' in /ROOT (126) has deleted/unused inode 20098 fid=[0xd10b000002000000:0x1oleg322-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (1650688, 265) != expected (1667072, 350)
oleg322-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (1650688, 265) != expected (1667072, 350)
oleg322-server: [QUOTA WARNING] Usage inconsistent for ID 0:actual (1650688, 265) != expected (1667072, 350)
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 4
000000:0x0].  Clear? no

check_dir_block:1649: increase inode 129 badness 3 to 4 for 20047
Entry 'MDT0000' in /ROOT/.lustre/lost+found (129) references inode 20105 fid=[0x200000403:0x1:0x0] found in group 1's unused inodes area.
Fix? no

Entry 'MDT0000' in /ROOT/.lustre/lost+found (129) has deleted/unused inode 20105 fid=[0x304000002000000:0x1000000:0x0].  Clear? no

Pass 2: Memory used: 264k/0k (100k/165k), time:  0.00/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 0MB, rate: 283.53MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 264k/0k (100k/165k), time:  0.03/ 0.01/ 0.00
Pass 3: Memory used: 264k/0k (98k/167k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 4: Checking reference counts
Inode 126 ref count is 4, should be 3.  Fix? no

Inode 129 ref count is 3, should be 2.  Fix? no

Pass 4: Memory used: 264k/0k (82k/182k), time:  0.00/ 0.00/ 0.00
Pass 4: I/O read: 1MB, write: 0MB, rate: 231.86MB/s
Pass 5: Checking group summary information
Block bitmap differences:  -12159 -(19840--19846)
Fix? no

Free blocks count wrong for group #0 (7745, counted=7737).
Fix? no

Free blocks count wrong for group #1 (4753, counted=4769).
Fix? no

Free blocks count wrong (12908, counted=12506).
Fix? no

Inode bitmap differences:  -(181--270) -(20098--20105)
Fix? no

Free inodes count wrong for group #0 (19820, counted=19732).
Fix? no

Free inodes count wrong for group #1 (19903, counted=19895).
Fix? no

Free inodes count wrong (39986, counted=39627).
Fix? no

Pass 5: Memory used: 376k/0k (82k/295k), time:  0.02/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 64.75MB/s
Update quota info for quota type 0? no

Update quota info for quota type 1? no

Update quota info for quota type 2? no


lustre-MDT0000: ********** WARNING: Filesystem still has errors **********


          14 inodes used (0.04%, out of 40000)
           2 non-contiguous files (14.3%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 203
       12092 blocks used (48.37%, out of 25000)
           0 bad blocks
           1 large file

         148 regular files
         117 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
         267 files
Memory used: 376k/0k (84k/293k), time:  0.06/ 0.01/ 0.00
I/O read: 1MB, write: 0MB, rate: 18.07MB/s
 sanity-lfsck test_2d: @@@@@@ FAIL: (2) Detected inconsistency on mds1 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7487:error()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:129:run_e2fsck_on_mds_facet()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:503:test_2d()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7863:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7741:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:515:main()
Dumping lctl log to /tmp/testlogs//sanity-lfsck.test_2d.*.1767317427.log
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_2d.debug_log.oleg322-server.1767317427.log.mmq8a0" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_2d.dmesg.oleg322-server.1767317427.log.CrJchC" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]
FAIL 2d (203s)

== sanity-lfsck test 2e: namespace LFSCK can verify remote object linkEA ========================================================== 20:30:32 (1767317432)
Checking servers environments
Checking clients oleg322-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg322-server'
oleg322-server: oleg322-server.virtnet: executing load_modules_local
oleg322-server: Loading modules from /home/green/git/lustre-release/lustre
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: detected 4 online CPUs by sysfs
oleg322-server: MODOPTS_LIBCFS=
oleg322-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Starting client: oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Starting client oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Started clients oleg322-client.virtnet: 
192.168.203.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9a5486e6f800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9a5486e6f800.idle_timeout=debug
disable quota as required
fail_loc=0x1603
fail_loc=0
Started LFSCK on the device lustre-MDT0000: scrub namespace
PASS 2e (177s)

== sanity-lfsck test 3: LFSCK can verify multiple-linked objects ========================================================== 20:33:29 (1767317609)
preparing... 4 * 4 files will be created Thu Jan  1 20:33:31 EST 2026.
total: 4 mkdir in 0.04 seconds: 109.30 ops/second
total: 4 create in 0.04 seconds: 106.20 ops/second
total: 4 mkdir in 0.03 seconds: 133.60 ops/second
prepared Thu Jan  1 20:33:35 EST 2026.
fail_loc=0x1603
fail_loc=0x1604
fail_loc=0
Started LFSCK on the device lustre-MDT0000: scrub namespace
PASS 3 (19s)

== sanity-lfsck test 4: FID-in-dirent can be rebuilt after MDT file-level backup/restore ========================================================== 20:33:48 (1767317628)
preparing... 3 * 3 files will be created Thu Jan  1 20:33:51 EST 2026.
total: 3 mkdir in 0.02 seconds: 140.40 ops/second
total: 3 create in 0.03 seconds: 103.24 ops/second
total: 3 mkdir in 0.02 seconds: 151.12 ops/second
prepared Thu Jan  1 20:33:55 EST 2026.
Stopping clients: oleg322-client.virtnet /mnt/lustre (opts:)
Stopping client oleg322-client.virtnet /mnt/lustre opts:
file-level backup/restore on mds1:/dev/vdc
oleg322-server: mount: /mnt/lustre-brpt: /dev/vdc already mounted or mount point busy.
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 32
 sanity-lfsck test_4: @@@@@@ FAIL: (1) Fail to backup/restore! 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7487:error()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:602:test_4()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7863:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7741:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:650:main()
Dumping lctl log to /tmp/testlogs//sanity-lfsck.test_4.*.1767317673.log
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_4.debug_log.oleg322-server.1767317673.log.qFavoP" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_4.dmesg.oleg322-server.1767317673.log.trJUBc" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]
FAIL 4 (50s)

== sanity-lfsck test 5: LFSCK can handle IGIF object upgrading ========================================================== 20:34:39 (1767317679)
Checking servers environments
Checking clients oleg322-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg322-server'
oleg322-server: oleg322-server.virtnet: executing load_modules_local
oleg322-server: Loading modules from /home/green/git/lustre-release/lustre
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: detected 4 online CPUs by sysfs
oleg322-server: MODOPTS_LIBCFS=
oleg322-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Starting client oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Started clients oleg322-client.virtnet: 
192.168.203.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9a54873bb800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9a54873bb800.idle_timeout=debug
disable quota as required
preparing... 1 * 1 files will be created Thu Jan  1 20:37:04 EST 2026.
fail_loc=0x1504
total: 1 mkdir in 0.02 seconds: 59.83 ops/second
total: 1 create in 0.02 seconds: 56.29 ops/second
total: 1 mkdir in 0.01 seconds: 71.16 ops/second
fail_loc=0
prepared Thu Jan  1 20:37:11 EST 2026.
Stopping clients: oleg322-client.virtnet /mnt/lustre (opts:)
Stopping client oleg322-client.virtnet /mnt/lustre opts:
file-level backup/restore on mds1:/dev/vdc
oleg322-server: mount: /mnt/lustre-brpt: /dev/vdc already mounted or mount point busy.
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 32
 sanity-lfsck test_5: @@@@@@ FAIL: (1) Fail to backup/restore! 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7487:error()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:661:test_5()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7863:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7741:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:716:main()
Dumping lctl log to /tmp/testlogs//sanity-lfsck.test_5.*.1767317845.log
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_5.debug_log.oleg322-server.1767317845.log.gDp7Kb" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_5.dmesg.oleg322-server.1767317845.log.V5t1lq" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]
FAIL 5 (172s)

== sanity-lfsck test 6a: LFSCK resumes from last checkpoint (1) ========================================================== 20:37:30 (1767317850)
Checking servers environments
Checking clients oleg322-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg322-server'
oleg322-server: oleg322-server.virtnet: executing load_modules_local
oleg322-server: Loading modules from /home/green/git/lustre-release/lustre
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: detected 4 online CPUs by sysfs
oleg322-server: MODOPTS_LIBCFS=
oleg322-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
Starting client: oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Starting client oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Started clients oleg322-client.virtnet: 
192.168.203.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9a5484106800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9a5484106800.idle_timeout=debug
disable quota as required
preparing... 5 * 5 files will be created Thu Jan  1 20:40:21 EST 2026.
total: 5 mkdir in 0.03 seconds: 144.51 ops/second
total: 5 create in 0.03 seconds: 147.37 ops/second
total: 5 mkdir in 0.04 seconds: 137.84 ops/second
prepared Thu Jan  1 20:40:26 EST 2026.
fail_val=1
fail_loc=0x1600
Started LFSCK on the device lustre-MDT0000: scrub namespace
fail_loc=0x80001608
fail_val=1
fail_loc=0x1600
Started LFSCK on the device lustre-MDT0000: scrub namespace
fail_loc=0
fail_val=0
PASS 6a (197s)

== sanity-lfsck test 6b: LFSCK resumes from last checkpoint (2) ========================================================== 20:40:47 (1767318047)
preparing... 5 * 5 files will be created Thu Jan  1 20:40:50 EST 2026.
total: 5 mkdir in 0.03 seconds: 173.51 ops/second
total: 5 create in 0.03 seconds: 185.13 ops/second
total: 5 mkdir in 0.05 seconds: 106.07 ops/second
prepared Thu Jan  1 20:40:55 EST 2026.
fail_val=1
fail_loc=0x1601
Started LFSCK on the device lustre-MDT0000: scrub namespace
fail_loc=0x80001609
fail_val=1
fail_loc=0x1601
Started LFSCK on the device lustre-MDT0000: scrub namespace
Additional debug for 6b
name: lfsck_namespace
magic: 0xa06249ff
version: 2
status: scanning-phase1
flags:
param:
last_completed_time: 1767318040
time_since_last_completed: 33 seconds
latest_start_time: 1767318069
time_since_latest_start: 4 seconds
last_checkpoint_time: 1767318064
time_since_last_checkpoint: 9 seconds
latest_start_position: 20100, [0x200005221:0x78:0x0], 0x60df4905479e5da
last_checkpoint_position: 20098, [0x200005221:0x78:0x0], 0x5d766c1a4819685
first_failure_position: N/A, N/A, N/A
checked_phase1: 8
checked_phase2: 0
updated_phase1: 0
updated_phase2: 0
failed_phase1: 0
failed_phase2: 0
directories: 2
dirent_repaired: 0
linkea_repaired: 0
nlinks_repaired: 0
multiple_linked_checked: 0
multiple_linked_repaired: 0
unknown_inconsistency: 0
unmatched_pairs_repaired: 0
dangling_repaired: 0
multiple_referenced_repaired: 0
bad_file_type_repaired: 0
lost_dirent_repaired: 0
local_lost_found_scanned: 0
local_lost_found_moved: 0
local_lost_found_skipped: 0
local_lost_found_failed: 0
striped_dirs_scanned: 0
striped_dirs_repaired: 0
striped_dirs_failed: 0
striped_dirs_disabled: 0
striped_dirs_skipped: 0
striped_shards_scanned: 0
striped_shards_repaired: 0
striped_shards_failed: 0
striped_shards_skipped: 0
name_hash_repaired: 0
linkea_overflow_cleared: 0
agent_entries_repaired: 0
success_count: 12
run_time_phase1: 11 seconds
run_time_phase2: 0 seconds
average_speed_phase1: 0 items/sec
average_speed_phase2: N/A
average_speed_total: 0 items/sec
real_time_speed_phase1: 0 items/sec
real_time_speed_phase2: N/A
current_position: 20099, [0x200005221:0x78:0x0], 0x8dd709c83e8a7ba
fail_loc=0
fail_val=0
PASS 6b (33s)

== sanity-lfsck test 7a: non-stopped LFSCK should auto restarts after MDS remount (1) ========================================================== 20:41:20 (1767318080)
preparing... 5 * 5 files will be created Thu Jan  1 20:41:24 EST 2026.
total: 5 mkdir in 0.03 seconds: 180.23 ops/second
total: 5 create in 0.03 seconds: 150.51 ops/second
total: 5 mkdir in 0.04 seconds: 121.29 ops/second
prepared Thu Jan  1 20:41:28 EST 2026.
192.168.203.122@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg322-client.virtnet /mnt/lustre (opts:)
fail_val=1
fail_loc=0x1601
Started LFSCK on the device lustre-MDT0000: scrub namespace
stop mds1
fail_loc=0
fail_val=0
start mds1
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
PASS 7a (74s)

== sanity-lfsck test 7b: non-stopped LFSCK should auto restarts after MDS remount (2) ========================================================== 20:42:34 (1767318154)
Checking servers environments
Checking clients oleg322-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg322-server'
oleg322-server: oleg322-server.virtnet: executing load_modules_local
oleg322-server: Loading modules from /home/green/git/lustre-release/lustre
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: detected 4 online CPUs by sysfs
oleg322-server: MODOPTS_LIBCFS=
oleg322-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Starting client: oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Starting client oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Started clients oleg322-client.virtnet: 
192.168.203.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9a549890c800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9a549890c800.idle_timeout=debug
disable quota as required
preparing... 2 * 2 files will be created Thu Jan  1 20:45:01 EST 2026.
total: 2 mkdir in 0.02 seconds: 130.34 ops/second
total: 2 create in 0.01 seconds: 161.12 ops/second
total: 2 mkdir in 0.02 seconds: 128.40 ops/second
prepared Thu Jan  1 20:45:08 EST 2026.
fail_loc=0x1604
fail_val=1
fail_loc=0x1602
Started LFSCK on the device lustre-MDT0000: scrub namespace
192.168.203.122@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg322-client.virtnet /mnt/lustre (opts:)
stop mds1
fail_loc=0
fail_val=0
start mds1
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
PASS 7b (195s)

== sanity-lfsck test 8: LFSCK state machine ============== 20:45:49 (1767318349)
formatall
oleg322-server: oleg322-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: oleg322-server.virtnet: executing load_modules_local
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
setupall
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: oleg322-server.virtnet: executing load_modules_local
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Using TIMEOUT=20
preparing... 20 * 20 files will be created Thu Jan  1 20:51:07 EST 2026.
total: 20 mkdir in 0.32 seconds: 63.22 ops/second
total: 20 create in 0.25 seconds: 78.74 ops/second
total: 20 mkdir in 0.25 seconds: 81.57 ops/second
prepared Thu Jan  1 20:51:16 EST 2026.
fail_loc=0x1603
fail_loc=0x1604
192.168.203.122@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg322-client.virtnet /mnt/lustre (opts:)
fail_val=2
fail_loc=0x1601
Started LFSCK on the device lustre-MDT0000: scrub namespace
Stopped LFSCK on the device lustre-MDT0000.
Started LFSCK on the device lustre-MDT0000: scrub namespace
fail_loc=0x80001609
Waiting 32s for 'failed'
Updated after 2s: want 'failed' got 'failed'
fail_loc=0x1600
Started LFSCK on the device lustre-MDT0000: scrub namespace
fail_loc=0x160a
stop mds1
fail_loc=0x160b
start mds1
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
fail_loc=0x1601
Started LFSCK on the device lustre-MDT0000: scrub namespace
stop mds1
fail_loc=0x160b
start mds1
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
stop mds1
start mds1 without resume LFSCK
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
fail_val=2
fail_loc=0x1602
Started LFSCK on the device lustre-MDT0000: scrub namespace
fail_loc=0
fail_val=0
Waiting 32s for 'completed'
Updated after 2s: want 'completed' got 'completed'
PASS 8 (466s)

== sanity-lfsck test 9a: LFSCK speed control (1) ========= 20:53:35 (1767318815)
Checking servers environments
Checking clients oleg322-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg322-server'
oleg322-server: oleg322-server.virtnet: executing load_modules_local
oleg322-server: Loading modules from /home/green/git/lustre-release/lustre
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: detected 4 online CPUs by sysfs
oleg322-server: MODOPTS_LIBCFS=
oleg322-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
Start of /dev/mapper/mds1_flakey on mds1 failed 17
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
Start of /dev/mapper/mds2_flakey on mds2 failed 17
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of /dev/mapper/ost2_flakey on ost2 failed 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
Starting client: oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Starting client oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Started clients oleg322-client.virtnet: 
192.168.203.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9a5493394800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9a5493394800.idle_timeout=debug
disable quota as required
 - open/close 951 (time 1767318992.52 total 10.00 last 95.07)
 - open/close 1686 (time 1767319002.54 total 20.03 last 73.31)
 - open/close 2330 (time 1767319012.57 total 30.06 last 64.21)
 - open/close 2855 (time 1767319022.58 total 40.07 last 52.44)
 - open/close 3433 (time 1767319032.59 total 50.08 last 57.76)
 - open/close 3908 (time 1767319042.61 total 60.09 last 47.42)
 - open/close 4401 (time 1767319052.61 total 70.09 last 49.29)
total: 5000 open/close in 77.87 seconds: 64.21 ops/second
Started LFSCK on the device lustre-MDT0000: scrub layout
PASS 9a (283s)

== sanity-lfsck test 9b: LFSCK speed control (2) ========= 20:58:18 (1767319098)
preparing... 0 * 0 files will be created Thu Jan  1 20:59:02 EST 2026.
prepared Thu Jan  1 20:59:07 EST 2026.
Preparing another 50 * 50 files (with error) at Thu Jan  1 20:59:07 EST 2026.
fail_loc=0x1604
total: 50 mkdir in 0.56 seconds: 89.38 ops/second
total: 50 create in 0.48 seconds: 104.24 ops/second
fail_loc=0x160c
Started LFSCK on the device lustre-MDT0000: scrub namespace
fail_loc=0
Prepared at Thu Jan  1 20:59:36 EST 2026.
Started LFSCK on the device lustre-MDT0000: scrub namespace
PASS 9b (111s)

== sanity-lfsck test 10: System is available during LFSCK scanning ========================================================== 21:00:09 (1767319209)
preparing... 1 * 1 files will be created Thu Jan  1 21:00:52 EST 2026.
total: 1 mkdir in 0.01 seconds: 143.37 ops/second
total: 1 create in 0.01 seconds: 188.39 ops/second
total: 1 mkdir in 0.01 seconds: 145.39 ops/second
prepared Thu Jan  1 21:00:57 EST 2026.
Preparing more files with error at Thu Jan  1 21:00:57 EST 2026.
fail_loc=0x1603
fail_loc=0x1604
fail_loc=0
Prepared at Thu Jan  1 21:03:20 EST 2026.
192.168.203.122@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg322-client.virtnet /mnt/lustre (opts:)
Starting client: oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Started LFSCK on the device lustre-MDT0000: scrub namespace
Waiting 90s for 'completed'
Waiting 50s for 'completed'
Waiting 40s for 'completed'
Updated after 70s: want 'completed' got 'completed'
PASS 10 (368s)

== sanity-lfsck test 11a: LFSCK can rebuild lost last_id ========================================================== 21:06:17 (1767319577)
total: 64 open/close in 1.13 seconds: 56.40 ops/second
stopall
remove LAST_ID on ost1: idx=0
removed '/mnt/lustre-ost1/O/0/LAST_ID'
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
fail_val=3
fail_loc=0x160e
trigger LFSCK for layout on ost1 to rebuild the LAST_ID(s)
Started LFSCK on the device lustre-OST0000: scrub layout
fail_val=0
fail_loc=0
the LAST_ID(s) should have been rebuilt
PASS 11a (275s)

== sanity-lfsck test 11b: LFSCK can rebuild crashed last_id ========================================================== 21:10:53 (1767319853)
Checking servers environments
Checking clients oleg322-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg322-server'
oleg322-server: oleg322-server.virtnet: executing load_modules_local
oleg322-server: Loading modules from /home/green/git/lustre-release/lustre
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: detected 4 online CPUs by sysfs
oleg322-server: MODOPTS_LIBCFS=
oleg322-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Started lustre-MDT0000
Started lustre-MDT0001
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
oleg322-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of /dev/mapper/ost1_flakey on ost1 failed 17
seq.cli-lustre-OST0001-super.width=65536
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Started lustre-OST0001
/mnt/lustre-ost2: 70 MiB (73404416 bytes) trimmed
Starting client: oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Starting client oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Started clients oleg322-client.virtnet: 
192.168.203.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9a549247e000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9a549247e000.idle_timeout=debug
disable quota as required
set fail_loc=0x160d to skip the updating LAST_ID on-disk
fail_loc=0x160d
 - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 3329 prealloc_next_id: 3298 count: 32
total: 64 open/close in 0.93 seconds: 68.49 ops/second
192.168.203.122@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg322-client.virtnet /mnt/lustre (opts:)
Stopping /mnt/lustre-ost1 (opts:) on oleg322-server
fail_loc=0x215
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Started lustre-OST0000
/mnt/lustre-ost1: 0 B (0 bytes) trimmed
the on-disk LAST_ID should be smaller than the expected one
trigger LFSCK for layout on ost1 to rebuild the on-disk LAST_ID
Started LFSCK on the device lustre-OST0000: scrub layout
Stopping /mnt/lustre-ost1 (opts:) on oleg322-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Started lustre-OST0000
the on-disk LAST_ID should have been rebuilt
fail_loc=0
Stopping clients: oleg322-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg322-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds2 (opts:-f) on oleg322-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg322-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg322-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg322-server
PASS 11b (261s)

== sanity-lfsck test 12a: single command to trigger LFSCK on all devices ========================================================== 21:15:14 (1767320114)
Checking servers environments
Checking clients oleg322-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg322-server'
oleg322-server: oleg322-server.virtnet: executing load_modules_local
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: Loading modules from /home/green/git/lustre-release/lustre
oleg322-server: detected 4 online CPUs by sysfs
oleg322-server: MODOPTS_LIBCFS=
oleg322-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Started lustre-MDT0000
Started lustre-MDT0001
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
seq.cli-lustre-OST0000-super.width=65536
seq.cli-lustre-OST0001-super.width=65536
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Started lustre-OST0000
Started lustre-OST0001
/mnt/lustre-ost1: 0 B (0 bytes) trimmed
/mnt/lustre-ost2: 0 B (0 bytes) trimmed
Starting client: oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Starting client oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Started clients oleg322-client.virtnet: 
192.168.203.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9a5493394000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9a5493394000.idle_timeout=debug
disable quota as required
total: 100 open/close in 1.73 seconds: 57.83 ops/second
total: 100 open/close in 1.61 seconds: 62.17 ops/second
Start namespace LFSCK on all targets by single command (-s 1).
Started LFSCK on the device lustre-MDT0000: scrub namespace
All the LFSCK targets should be in 'scanning-phase1' status.
Stop namespace LFSCK on all targets by single lctl command.
Stopped LFSCK on the device lustre-MDT0000.
All the LFSCK targets should be in 'stopped' status.
Re-start namespace LFSCK on all targets by single command (-s 0).
Started LFSCK on the device lustre-MDT0000: scrub namespace
All the LFSCK targets should be in 'completed' status.
debug=-1
debug_mb=150
debug=-1
debug_mb=150
Start layout LFSCK on all targets by single command (-s 1).
Started LFSCK on the device lustre-MDT0000: scrub layout
All the LFSCK targets should be in 'scanning-phase1' status.
Stop layout LFSCK on all targets by single lctl command.
Stopped LFSCK on the device lustre-MDT0000.
All the LFSCK targets should be in 'stopped' status.
oleg322-server: error: get_param: param_path 'obdfilter//lfsck_layout': No such file or directory
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 2
 sanity-lfsck test_12a: @@@@@@ FAIL: (12) OST1 Expect 'stopped', but got '' 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7487:error()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:1543:test_12a()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7863:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7741:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:1555:main()
Dumping lctl log to /tmp/testlogs//sanity-lfsck.test_12a.*.1767320312.log
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_12a.debug_log.oleg322-server.1767320312.log.Yg397c" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_12a.dmesg.oleg322-server.1767320312.log.UHxQyT" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]
FAIL 12a (207s)

== sanity-lfsck test 12b: auto detect Lustre device ====== 21:18:41 (1767320321)
Start LFSCK without '-M' specified.
Started LFSCK on the device lustre-MDT0000: scrub layout namespace

Start layout LFSCK on the node with multipe targets,
but not specify '-M'/'-A' option. Should get failure.

oleg322-server: Detect multiple devices on current node. Please specify the device explicitly via '-M' option or '-A' option for all.
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 22
PASS 12b (17s)

== sanity-lfsck test 13: LFSCK can repair crashed lmm_oi ========================================================== 21:18:58 (1767320338)
#####
The lmm_oi in layout EA should be consistent with the MDT-object
FID; otherwise, the LFSCK should re-generate the lmm_oi from the
MDT-object FID.
#####
Inject failure stub to simulate bad lmm_oi
fail_loc=0x160f
total: 1 open/close in 0.04 seconds: 27.02 ops/second
fail_loc=0
Trigger layout LFSCK to find out the bad lmm_oi and fix them
Started LFSCK on the device lustre-MDT0000: scrub layout
PASS 13 (13s)

== sanity-lfsck test 14a: LFSCK can repair MDT-object with dangling LOV EA reference (1) ========================================================== 21:19:11 (1767320351)
#####
The OST-object referenced by the MDT-object should be there;
otherwise, the LFSCK should re-create the missing OST-object.
without '--delay-create-ostobj' option.
#####
Inject failure stub to simulate dangling referenced MDT-object
fail_loc=0x1610
 - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 3489 prealloc_next_id: 3445 count: 45
total: 61 open/close in 1.24 seconds: 49.14 ops/second
touch: setting times of '/mnt/lustre/d14a.sanity-lfsck/guard0': No such file or directory
touch: setting times of '/mnt/lustre/d14a.sanity-lfsck/guard1': No such file or directory
fail_loc=0
debug=-1
debug_mb=150
debug=-1
debug_mb=150
 - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 3553 prealloc_next_id: 3524 count: 30
total: 30 open/close in 0.81 seconds: 36.90 ops/second
'ls' should fail because of dangling referenced MDT-object
Trigger layout LFSCK to find out dangling reference
Started LFSCK on the device lustre-MDT0000: scrub layout
'stat' should fail because of not repair dangling by default
Trigger layout LFSCK to repair dangling reference
Started LFSCK on the device lustre-MDT0000: scrub layout
'stat' should success after layout LFSCK repairing
debug_mb=148
debug_mb=148
stopall to cleanup object cache
setupall
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: oleg322-server.virtnet: executing load_modules_local
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Using TIMEOUT=20
PASS 14a (263s)

== sanity-lfsck test 14b: LFSCK can repair MDT-object with dangling LOV EA reference (2) ========================================================== 21:23:35 (1767320615)
#####
The OST-object referenced by the MDT-object should be there;
otherwise, the LFSCK should re-create the missing OST-object.
with '--delay-create-ostobj' option.
#####
Inject failure stub to simulate dangling referenced MDT-object
fail_loc=0x1610
 - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 3617 prealloc_next_id: 3586 count: 32
total: 63 open/close in 1.25 seconds: 50.38 ops/second
touch: setting times of '/mnt/lustre/d14b.sanity-lfsck/guard': No such file or directory
fail_loc=0
debug=-1
debug_mb=150
debug=-1
debug_mb=150
 - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 3681 prealloc_next_id: 3650 count: 32
total: 32 open/close in 0.96 seconds: 33.29 ops/second
'ls' should fail because of dangling referenced MDT-object
Trigger layout LFSCK to find out dangling reference
Started LFSCK on the device lustre-MDT0000: scrub layout
'stat' should fail because of not repair dangling by default
Trigger layout LFSCK to repair dangling reference
Started LFSCK on the device lustre-MDT0000: scrub layout
'stat' should success after layout LFSCK repairing
debug_mb=148
debug_mb=148
stopall to cleanup object cache
setupall
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: oleg322-server.virtnet: executing load_modules_local
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Using TIMEOUT=20
PASS 14b (215s)

== sanity-lfsck test 15a: LFSCK can repair unmatched MDT-object/OST-object pairs (1) ========================================================== 21:27:09 (1767320829)
#####
If the OST-object referenced by the MDT-object back points
to some non-exist MDT-object, then the LFSCK should repair
the OST-object to back point to the right MDT-object.
#####
Inject failure stub to make the OST-object to back point to
non-exist MDT-object.
fail_loc=0x1611
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104029 s, 10.1 MB/s
257+0 records in
257+0 records out
1052672 bytes (1.1 MB, 1.0 MiB) copied, 0.422213 s, 2.5 MB/s
fail_loc=0
Trigger layout LFSCK to find out unmatched pairs and fix them
Started LFSCK on the device lustre-MDT0000: scrub layout
PASS 15a (18s)

== sanity-lfsck test 15b: LFSCK can repair unmatched MDT-object/OST-object pairs (2) ========================================================== 21:27:27 (1767320847)
#####
If the OST-object referenced by the MDT-object back points
to other MDT-object that doesn't recognize the OST-object,
then the LFSCK should repair it to back point to the right
MDT-object (the first one).
#####
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344584 s, 30.4 MB/s
Inject failure stub to make the OST-object to back point to
other MDT-object
fail_loc=0x1612
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478519 s, 21.9 MB/s
2+0 records in
2+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.108276 s, 19.4 MB/s
fail_loc=0
Trigger layout LFSCK to find out unmatched pairs and fix them
Started LFSCK on the device lustre-MDT0000: scrub layout
PASS 15b (12s)

== sanity-lfsck test 15c: LFSCK can repair unmatched MDT-object/OST-object pairs (3) ========================================================== 21:27:39 (1767320859)

 SKIP: sanity-lfsck test_15c MDS newer than 2.7.55, LU-6475
SKIP 15c (4s)

== sanity-lfsck test 15d: LFSCK don't crash upon dir migration failure ========================================================== 21:27:43 (1767320863)
total: 100 open/close in 1.56 seconds: 64.12 ops/second
total: 100 mkdir in 0.70 seconds: 142.91 ops/second
Migrate /mnt/lustre/d15d.sanity-lfsck to MDT1
fail_loc=0x1709
lfs migrate: /mnt/lustre/d15d.sanity-lfsck/f7 migrate failed: Input/output error (5)
lfs migrate: cb_migrate_mdt_fini: error completing migration of /mnt/lustre/d15d.sanity-lfsck: Directory not empty (39)
fail_loc=0
fail_loc=0x1709
fail_loc=0
Started LFSCK on the device lustre-MDT0000: scrub namespace
lfs migrate: cb_migrate_mdt_fini: error completing migration of /mnt/lustre/d15d.sanity-lfsck: Directory not empty (39)
lfs rm_entry: error on ioctl 0xc03066f0 for '*' (3): No such file or directory (2)
debug=0
Stopping clients: oleg322-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg322-client.virtnet /mnt/lustre2 (opts:)
Stopping client oleg322-client.virtnet /mnt/lustre opts:
Stopping /mnt/lustre-mds1 (opts:-f) on oleg322-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg322-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg322-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg322-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg322-server'
oleg322-server: oleg322-server.virtnet: executing unload_modules_local
oleg322-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg322-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.
=== sanity-lfsck: start setup 21:29:44 (1767320984) ===
Stopping clients: oleg322-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg322-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 2
oleg322-server: oleg322-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg322-server'
oleg322-server: oleg322-server.virtnet: executing load_modules_local
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: Loading modules from /home/green/git/lustre-release/lustre
oleg322-server: detected 4 online CPUs by sysfs
oleg322-server: MODOPTS_LIBCFS=
oleg322-server: Force libcfs to create 2 CPU partitions
oleg322-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg322-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Formatting mgs, mds, osts
Format ost1: /dev/vde
Format mds1: /dev/vdc
Format mds2: /dev/vdd
Format ost2: /dev/vdf
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Checking servers environments
Checking clients oleg322-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg322-server'
oleg322-server: oleg322-server.virtnet: executing load_modules_local
oleg322-server: Loading modules from /home/green/git/lustre-release/lustre
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: detected 4 online CPUs by sysfs
oleg322-server: MODOPTS_LIBCFS=
oleg322-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Commit the device label on /dev/vdd
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Commit the device label on /dev/vdc
Started lustre-MDT0000
Started lustre-MDT0001
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
seq.cli-lustre-OST0001-super.width=65536
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Commit the device label on /dev/vdf
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Commit the device label on /dev/vde
Started lustre-OST0000
Started lustre-OST0001
/mnt/lustre-ost1: 0 B (0 bytes) trimmed
/mnt/lustre-ost2: 0 B (0 bytes) trimmed
Starting client: oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Starting client oleg322-client.virtnet:  -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre
Started clients oleg322-client.virtnet: 
192.168.203.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9a54a0615800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9a54a0615800.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
Updated after 5s: want 'procname_uid' got 'procname_uid'
disable quota as required
osd-ldiskfs.track_declares_assert=1
=== sanity-lfsck: finish setup 21:34:52 (1767321292) ===
PASS 15d (436s)

== sanity-lfsck test 16: LFSCK can repair inconsistent MDT-object/OST-object owner ========================================================== 21:34:59 (1767321299)
#####
If the OST-object's owner information does not match the owner
information stored in the MDT-object, then the LFSCK trust the
MDT-object and update the OST-object's owner information.
#####
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733988 s, 14.3 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [createmany] [-o] [/mnt/lustre/d16.sanity-lfsck/d1/o] [100]
total: 100 open/close in 1.52 seconds: 65.71 ops/second
Inject failure stub to skip OST-object owner changing
fail_loc=0x1613
fail_loc=0
Trigger layout LFSCK to find out inconsistent OST-object owner
and fix them
Started LFSCK on the device lustre-MDT0000: scrub layout
PASS 16 (13s)

== sanity-lfsck test 17: LFSCK can repair multiple references ========================================================== 21:35:13 (1767321313)
#####
If more than one MDT-objects reference the same OST-object,
and the OST-object only recognizes one MDT-object, then the
LFSCK should create new OST-objects for such non-recognized
MDT-objects.
#####
Inject failure stub to make two MDT-objects to refernce
the OST-object
fail_val=0
fail_loc=0x1614
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669064 s, 15.7 MB/s
total: 1 open/close in 0.03 seconds: 39.71 ops/second
fail_loc=0
fail_val=0
/mnt/lustre/d17.sanity-lfsck/f0 and /mnt/lustre/d17.sanity-lfsck/guard use the same OST-objects
/mnt/lustre/d17.sanity-lfsck/f1 and /mnt/lustre/d17.sanity-lfsck/guard use the same OST-objects
Trigger layout LFSCK to find out multiple refenced MDT-objects
and fix them
Started LFSCK on the device lustre-MDT0000: scrub layout
/mnt/lustre/d17.sanity-lfsck/f0 and /mnt/lustre/d17.sanity-lfsck/guard should use diff OST-objects
2+0 records in
2+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.102301 s, 20.5 MB/s
/mnt/lustre/d17.sanity-lfsck/f1 and /mnt/lustre/d17.sanity-lfsck/guard should use diff OST-objects
2+0 records in
2+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.053317 s, 39.3 MB/s
PASS 17 (12s)

== sanity-lfsck test 18a: Find out orphan OST-object and repair it (1) ========================================================== 21:35:25 (1767321325)
#####
The target MDT-object is there, but related stripe information
is lost or partly lost. The LFSCK should regenerate the missing
layout EA entries.
#####
2+0 records in
2+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.06752 s, 31.1 MB/s
[0x200000402:0x73:0x0]
/mnt/lustre/d18a.sanity-lfsck/a1/f1
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	           108	         0x6c	   0x2c0000400

2+0 records in
2+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.126937 s, 16.5 MB/s
[0x240000401:0x2:0x0]
/mnt/lustre/d18a.sanity-lfsck/a2/f2
lmm_stripe_count:  2
lmm_stripe_size:   1048576
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 1
	obdidx		 objid		 objid		 group
	     1	             2	          0x2	   0x280000401
	     0	             2	          0x2	   0x2c0000401

2+0 records in
2+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.120158 s, 17.5 MB/s
[0x200000402:0x75:0x0]
/mnt/lustre/d18a.sanity-lfsck/f3
  lcm_layout_gen:    3
  lcm_mirror_count:  1
  lcm_entry_count:   2
    lcme_id:             1
    lcme_mirror_id:      0
    lcme_flags:          init
    lcme_extent.e_start: 0
    lcme_extent.e_end:   1048576
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: 0
      lmm_objects:
      -   0: { l_ost_idx:   0, l_fid: [0x2c0000400:0x6d:0x0] }

    lcme_id:             2
    lcme_mirror_id:      0
    lcme_flags:          init
    lcme_extent.e_start: 1048576
    lcme_extent.e_end:   EOF
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: 1
      lmm_objects:
      -   0: { l_ost_idx:   1, l_fid: [0x280000400:0x2:0x0] }

Inject failure, to make the MDT-object lost its layout EA
fail_loc=0x1615
fail_loc=0x1615
fail_loc=0
fail_loc=0
The file size should be incorrect since layout EA is lost
Trigger layout LFSCK on all devices to find out orphan OST-object
Started LFSCK on the device lustre-MDT0000: scrub layout
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Waiting 120s for 'completed'
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Waiting 90s for 'completed'
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Waiting 70s for 'completed'
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Waiting 20s for 'completed'
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Waiting 0s for 'completed'
Update not seen after 120s: want 'completed' got ''
 sanity-lfsck test_18a: @@@@@@ FAIL: (4) MDS1 is not the expected 'completed' 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7487:error()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:2207:test_18a()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7863:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7741:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:2258:main()
Dumping lctl log to /tmp/testlogs//sanity-lfsck.test_18a.*.1767321459.log
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_18a.debug_log.oleg322-server.1767321459.log.ytJxmF" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_18a.dmesg.oleg322-server.1767321459.log.8dFC8R" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]
FAIL 18a (137s)

 SKIP: sanity-lfsck test_18b skipping excluded test 18b

== sanity-lfsck test 18c: Find out orphan OST-object and repair it (3) ========================================================== 21:37:44 (1767321464)
#####
The target MDT-object is lost, and the OST-object FID is missing.
The LFSCK should re-create the MDT-object with new FID under the 
directory .lustre/lost+found/MDTxxxx.
#####
Inject failure, to simulate the case of missing parent FID
fail_loc=0x1617
2+0 records in
2+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0902084 s, 23.2 MB/s
/mnt/lustre/d18c.sanity-lfsck/a1/f1
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	           110	         0x6e	   0x2c0000400

2+0 records in
2+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104515 s, 20.1 MB/s
/mnt/lustre/d18c.sanity-lfsck/a2/f2
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	             3	          0x3	   0x2c0000401

2+0 records in
2+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0947793 s, 22.1 MB/s
/mnt/lustre/d18c.sanity-lfsck/f3
  lcm_layout_gen:    3
  lcm_mirror_count:  1
  lcm_entry_count:   2
    lcme_id:             1
    lcme_mirror_id:      0
    lcme_flags:          init
    lcme_extent.e_start: 0
    lcme_extent.e_end:   1048576
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: 0
      lmm_objects:
      -   0: { l_ost_idx:   0, l_fid: [0x2c0000400:0x6f:0x0] }

    lcme_id:             2
    lcme_mirror_id:      0
    lcme_flags:          init
    lcme_extent.e_start: 1048576
    lcme_extent.e_end:   EOF
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: 1
      lmm_objects:
      -   0: { l_ost_idx:   1, l_fid: [0x280000400:0x3:0x0] }

fail_loc=0
Inject failure, to simulate the case of missing the MDT-object
fail_loc=0x1616
fail_loc=0x1616
fail_loc=0
fail_loc=0
Trigger layout LFSCK on all devices to find out orphan OST-object
Started LFSCK on the device lustre-MDT0000: scrub layout
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Waiting 120s for 'completed'
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Waiting 110s for 'completed'
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Waiting 100s for 'completed'
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Waiting 90s for 'completed'
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Waiting 50s for 'completed'
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Waiting 40s for 'completed'
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Waiting 30s for 'completed'
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Waiting 10s for 'completed'
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Update not seen after 120s: want 'completed' got ''
 sanity-lfsck test_18c: @@@@@@ FAIL: (2) MDS1 is not the expected 'completed' 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7487:error()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:2493:test_18c()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7863:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7741:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:2541:main()
Dumping lctl log to /tmp/testlogs//sanity-lfsck.test_18c.*.1767321598.log
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_18c.debug_log.oleg322-server.1767321598.log.xEkryP" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_18c.dmesg.oleg322-server.1767321598.log.ZE3dy2" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]
FAIL 18c (138s)

== sanity-lfsck test 18d: Find out orphan OST-object and repair it (4) ========================================================== 21:40:02 (1767321602)
#####
The target MDT-object layout EA is corrupted, but the right
OST-object is still alive as orphan. The layout LFSCK will
not create new OST-object to occupy such slot.
#####
[0x200000402:0x81:0x0]
/mnt/lustre/d18d.sanity-lfsck/a1/f1
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	           112	         0x70	   0x2c0000400

[0x200000402:0x82:0x0]
/mnt/lustre/d18d.sanity-lfsck/a1/f2
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	           113	         0x71	   0x2c0000400

[0x200000402:0x83:0x0]
/mnt/lustre/d18d.sanity-lfsck/a1/f3
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	           114	         0x72	   0x2c0000400

[0x200000402:0x84:0x0]
/mnt/lustre/d18d.sanity-lfsck/a1/f4
  lcm_layout_gen:    2
  lcm_mirror_count:  1
  lcm_entry_count:   2
    lcme_id:             1
    lcme_mirror_id:      0
    lcme_flags:          init
    lcme_extent.e_start: 0
    lcme_extent.e_end:   1048576
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: 0
      lmm_objects:
      -   0: { l_ost_idx:   0, l_fid: [0x2c0000400:0x73:0x0] }

    lcme_id:             2
    lcme_mirror_id:      0
    lcme_flags:          0
    lcme_extent.e_start: 1048576
    lcme_extent.e_end:   EOF
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: -1

Inject failure to make /mnt/lustre/d18d.sanity-lfsck/a1/f1 and /mnt/lustre/d18d.sanity-lfsck/a1/f2
to reference the same OST-object (which is f1's OST-object).
Then drop /mnt/lustre/d18d.sanity-lfsck/a1/f1 and its OST-object, so f2 becomes
dangling reference case, but f2's old OST-object is there.
The failure also makes /mnt/lustre/d18d.sanity-lfsck/a1/f3 and /mnt/lustre/d18d.sanity-lfsck/a1/f4
to reference the same OST-object (which is f3's OST-object).
Then drop /mnt/lustre/d18d.sanity-lfsck/a1/f3 and its OST-object, so f4 becomes
dangling reference case, but f4's old OST-object is there.

fail_loc=0x1618
fail_loc=0
stopall to cleanup object cache
setupall
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: oleg322-server.virtnet: executing load_modules_local
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Using TIMEOUT=20
Trigger layout LFSCK on all devices to find out orphan OST-object
Started LFSCK on the device lustre-MDT0000: scrub layout
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Waiting 120s for 'completed'
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Waiting 110s for 'completed'
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Waiting 90s for 'completed'
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Waiting 70s for 'completed'
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Waiting 50s for 'completed'
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Waiting 30s for 'completed'
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Waiting 20s for 'completed'
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Update not seen after 120s: want 'completed' got ''
 sanity-lfsck test_18d: @@@@@@ FAIL: (3) MDS1 is not the expected 'completed' 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7487:error()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:2612:test_18d()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7863:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7741:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:2652:main()
Dumping lctl log to /tmp/testlogs//sanity-lfsck.test_18d.*.1767321939.log
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_18d.debug_log.oleg322-server.1767321939.log.vDAwfo" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_18d.dmesg.oleg322-server.1767321939.log.eoQJ2k" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]
FAIL 18d (341s)

== sanity-lfsck test 18e: Find out orphan OST-object and repair it (5) ========================================================== 21:45:43 (1767321943)
#####
The target MDT-object layout EA slot is occpuied by some new
created OST-object when repair dangling reference case. Such
conflict OST-object has been modified by others. To keep the
new data, the LFSCK will create a new file to refernece this
old orphan OST-object.
#####
[0x200000bd1:0x4:0x0]
/mnt/lustre/d18e.sanity-lfsck/a1/f1
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	           162	         0xa2	   0x2c0000400

[0x200000bd1:0x5:0x0]
/mnt/lustre/d18e.sanity-lfsck/a1/f2
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	           163	         0xa3	   0x2c0000400

[0x200000bd1:0x6:0x0]
/mnt/lustre/d18e.sanity-lfsck/a1/f3
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	           164	         0xa4	   0x2c0000400

[0x200000bd1:0x7:0x0]
/mnt/lustre/d18e.sanity-lfsck/a1/f4
  lcm_layout_gen:    2
  lcm_mirror_count:  1
  lcm_entry_count:   2
    lcme_id:             1
    lcme_mirror_id:      0
    lcme_flags:          init
    lcme_extent.e_start: 0
    lcme_extent.e_end:   1048576
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: 0
      lmm_objects:
      -   0: { l_ost_idx:   0, l_fid: [0x2c0000400:0xa5:0x0] }

    lcme_id:             2
    lcme_mirror_id:      0
    lcme_flags:          0
    lcme_extent.e_start: 1048576
    lcme_extent.e_end:   EOF
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: -1

Inject failure to make /mnt/lustre/d18e.sanity-lfsck/a1/f1 and /mnt/lustre/d18e.sanity-lfsck/a1/f2
to reference the same OST-object (which is f1's OST-object).
Then drop /mnt/lustre/d18e.sanity-lfsck/a1/f1 and its OST-object, so f2 becomes
dangling reference case, but f2's old OST-object is there.
Also the failure makes /mnt/lustre/d18e.sanity-lfsck/a1/f3 and /mnt/lustre/d18e.sanity-lfsck/a1/f4
to reference the same OST-object (which is f3's OST-object).
Then drop /mnt/lustre/d18e.sanity-lfsck/a1/f3 and its OST-object, so f4 becomes
dangling reference case, but f4's old OST-object is there.

fail_loc=0x1618
fail_loc=0
stopall to cleanup object cache
setupall
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: oleg322-server.virtnet: executing load_modules_local
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy
oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1
Using TIMEOUT=20
fail_val=10
fail_loc=0x1602
debug=-1
debug_mb=150
debug=-1
debug_mb=150
Trigger layout LFSCK on all devices to find out orphan OST-object
Started LFSCK on the device lustre-MDT0000: scrub layout
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Waiting 120s for 'scanning-phase2'
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Waiting 90s for 'scanning-phase2'
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Waiting 70s for 'scanning-phase2'
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Waiting 40s for 'scanning-phase2'
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Waiting 30s for 'scanning-phase2'
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Waiting 20s for 'scanning-phase2'
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
oleg322-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory
Update not seen after 120s: want 'scanning-phase2' got ''
 sanity-lfsck test_18e: @@@@@@ FAIL: (3) MDS1 is not the expected 'scanning-phase2' 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7487:error()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:2730:test_18e()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7863:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7741:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:2805:main()
Dumping lctl log to /tmp/testlogs//sanity-lfsck.test_18e.*.1767322282.log
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_18e.debug_log.oleg322-server.1767322282.log.4cCvNK" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-lfsck.test_18e.dmesg.oleg322-server.1767322282.log.0K073R" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]
FAIL 18e (346s)

== sanity-lfsck test 18f: Skip the failed OST(s) when handle orphan OST-objects ========================================================== 21:51:29 (1767322289)
#####
The target MDT-object is lost. The LFSCK should re-create the
MDT-object under .lustre/lost+found/MDTxxxx. If some OST fail
to verify some OST-object(s) during the first stage-scanning,
the LFSCK should skip orphan OST-objects for such OST. Others
should not be affected.
#####
2+0 records in
2+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0530485 s, 39.5 MB/s
2+0 records in
2+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0691477 s, 30.3 MB/s