-----============= acceptance-small: sanity-lfsck ============----- Thu Aug 27 14:57:57 EDT 2026
MGS_OS_ID_LIKE=rhel centos fedora rocky
MGS_OS_VERSION_ID=8.10
MGS_OS_ID=rocky
MGS_OS_VERSION_CODE=134873088
MDS1_OS_VERSION_ID=8.10
MDS1_OS_VERSION_CODE=134873088
MDS1_OS_ID_LIKE=rhel centos fedora rocky
MDS1_OS_ID=rocky
OST1_OS_VERSION_CODE=134873088
OST1_OS_ID_LIKE=rhel centos fedora rocky
OST1_OS_VERSION_ID=8.10
OST1_OS_ID=rocky
CLIENT_OS_ID=rocky
CLIENT_OS_VERSION_CODE=134873088
CLIENT_OS_VERSION_ID=8.10
CLIENT_OS_ID_LIKE=rhel centos fedora rocky
oleg420-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-lfsck.*ex': No such file or directory
excepting tests: 18b 23b
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg420-server'
oleg420-server: oleg420-server.virtnet: executing load_modules_local
oleg420-server: Loading modules from /home/green/git/lustre-release/lustre
oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg420-server: detected 4 online CPUs by sysfs
oleg420-server: MODOPTS_LIBCFS=
oleg420-server: Force libcfs to create 2 CPU partitions
client=34683217 MDS=34683217 OSS=34683217
Stopping clients: oleg420-client.virtnet /mnt/lustre (opts:)
Stopping client oleg420-client.virtnet /mnt/lustre opts:
Stopping clients: oleg420-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg420-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg420-server

unloading modules via unload_modules_local on: 'oleg420-server'
oleg420-server: oleg420-server.virtnet: executing unload_modules_local
oleg420-server: 
modules unloaded.
=== sanity-lfsck: start setup 14:59:13 (1787857153) ===
Stopping clients: oleg420-client.virtnet /mnt/lustre (opts:-f)
Stopping clients: oleg420-client.virtnet /mnt/lustre2 (opts:-f)
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 2
oleg420-server: oleg420-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg420-server'
oleg420-server: oleg420-server.virtnet: executing load_modules_local
oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg420-server: Loading modules from /home/green/git/lustre-release/lustre
oleg420-server: detected 4 online CPUs by sysfs
oleg420-server: MODOPTS_LIBCFS=
oleg420-server: Force libcfs to create 2 CPU partitions
oleg420-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg420-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg420-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Formatting mgs, mds, osts
Format mds1: lustre-mdt1/mdt1
Format ost1: lustre-ost1/ost1
Format ost2: lustre-ost2/ost2
Checking servers environments
Checking clients oleg420-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg420-server'
oleg420-server: oleg420-server.virtnet: executing load_modules_local
oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg420-server: Loading modules from /home/green/git/lustre-release/lustre
oleg420-server: detected 4 online CPUs by sysfs
oleg420-server: MODOPTS_LIBCFS=
oleg420-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
Commit the device label on lustre-mdt1/mdt1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
Commit the device label on lustre-ost1/ost1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
Commit the device label on lustre-ost2/ost2
Started lustre-OST0001
Starting client: oleg420-client.virtnet:  -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre
Starting client oleg420-client.virtnet:  -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre
Started clients oleg420-client.virtnet: 
192.168.204.120@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff96f79848f000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff96f79848f000.idle_timeout=debug
setting jobstats to procname_uid
Setting lustre.sys.jobid_var from disable to procname_uid
Waiting 90s for 'procname_uid'
disable quota as required
=== sanity-lfsck: finish setup 15:01:48 (1787857308) ===

== sanity-lfsck test 0: Control LFSCK manually =========== 15:01:52 (1787857312)
preparing... 3 * 3 files will be created Thu Aug 27 15:01:53 EDT 2026.
total: 3 mkdir in 0.02 seconds: 144.42 ops/second
total: 3 create in 0.03 seconds: 115.00 ops/second
total: 3 mkdir in 0.02 seconds: 135.73 ops/second
prepared Thu Aug 27 15:01:58 EDT 2026.
fail_val=3
fail_loc=0x1600
Started LFSCK on the device lustre-MDT0000: scrub namespace
name: lfsck_namespace
magic: 0xa06249ff
version: 2
status: scanning-phase1
flags:
param:
last_completed_time: N/A
time_since_last_completed: N/A
latest_start_time: 1787857320
time_since_latest_start: 1 seconds
last_checkpoint_time: N/A
time_since_last_checkpoint: N/A
latest_start_position: 310, N/A, N/A
last_checkpoint_position: N/A, N/A, N/A
first_failure_position: N/A, N/A, N/A
checked_phase1: 0
checked_phase2: 0
updated_phase1: 0
updated_phase2: 0
failed_phase1: 0
failed_phase2: 0
directories: 0
dirent_repaired: 0
linkea_repaired: 0
nlinks_repaired: 0
multiple_linked_checked: 0
multiple_linked_repaired: 0
unknown_inconsistency: 0
unmatched_pairs_repaired: 0
dangling_repaired: 0
multiple_referenced_repaired: 0
bad_file_type_repaired: 0
lost_dirent_repaired: 0
local_lost_found_scanned: 0
local_lost_found_moved: 0
local_lost_found_skipped: 0
local_lost_found_failed: 0
striped_dirs_scanned: 0
striped_dirs_repaired: 0
striped_dirs_failed: 0
striped_dirs_disabled: 0
striped_dirs_skipped: 0
striped_shards_scanned: 0
striped_shards_repaired: 0
striped_shards_failed: 0
striped_shards_skipped: 0
name_hash_repaired: 0
linkea_overflow_cleared: 0
agent_entries_repaired: 0
success_count: 0
run_time_phase1: 1 seconds
run_time_phase2: 0 seconds
average_speed_phase1: 0 items/sec
average_speed_phase2: N/A
average_speed_total: 0 items/sec
real_time_speed_phase1: 0 items/sec
real_time_speed_phase2: N/A
current_position: 309, N/A, N/A
Stopped LFSCK on the device lustre-MDT0000.
Started LFSCK on the device lustre-MDT0000: scrub namespace
fail_loc=0
fail_val=0
Waiting 32s for 'completed'
Updated after 2s: want 'completed' got 'completed'
Started LFSCK on the device lustre-MDT0000: scrub namespace
stopall, should NOT crash LU-3649
Stopping clients: oleg420-client.virtnet /mnt/lustre (opts:)
Stopping client oleg420-client.virtnet /mnt/lustre opts:
Stopping clients: oleg420-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg420-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg420-server
PASS 0 (57s)

== sanity-lfsck test 1a: LFSCK can find out and repair crashed FID-in-dirent ========================================================== 15:02:49 (1787857369)
Checking servers environments
Checking clients oleg420-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg420-server'
oleg420-server: oleg420-server.virtnet: executing load_modules_local
oleg420-server: Loading modules from /home/green/git/lustre-release/lustre
oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg420-server: detected 4 online CPUs by sysfs
oleg420-server: MODOPTS_LIBCFS=
oleg420-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
Started lustre-OST0001
Starting client: oleg420-client.virtnet:  -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre
Starting client oleg420-client.virtnet:  -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre
Started clients oleg420-client.virtnet: 
192.168.204.120@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff96f7896c2800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff96f7896c2800.idle_timeout=debug
disable quota as required
preparing... 1 * 1 files will be created Thu Aug 27 15:04:00 EDT 2026.
total: 1 mkdir in 0.01 seconds: 169.67 ops/second
total: 1 create in 0.01 seconds: 108.54 ops/second
total: 1 mkdir in 0.01 seconds: 152.93 ops/second
prepared Thu Aug 27 15:04:05 EDT 2026.
fail_loc=0x1501
fail_loc=0
192.168.204.120@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg420-client.virtnet /mnt/lustre (opts:)
Started LFSCK on the device lustre-MDT0000: scrub namespace
Starting client: oleg420-client.virtnet:  -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre
fail_loc=0x1505
fail_loc=0
PASS 1a (88s)

== sanity-lfsck test 1b: LFSCK can find out and repair the missing FID-in-LMA ========================================================== 15:04:17 (1787857457)

 SKIP: sanity-lfsck test_1b OI Scrub not implemented for ZFS
SKIP 1b (3s)

== sanity-lfsck test 1c: LFSCK can find out and repair lost FID-in-dirent ========================================================== 15:04:20 (1787857460)
preparing... 1 * 1 files will be created Thu Aug 27 15:04:23 EDT 2026.
total: 1 mkdir in 0.01 seconds: 146.11 ops/second
total: 1 create in 0.01 seconds: 132.87 ops/second
total: 1 mkdir in 0.00 seconds: 237.25 ops/second
prepared Thu Aug 27 15:04:27 EDT 2026.
fail_loc=0x1504
fail_loc=0
192.168.204.120@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg420-client.virtnet /mnt/lustre (opts:)
Started LFSCK on the device lustre-MDT0000: scrub namespace
Starting client: oleg420-client.virtnet:  -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre
fail_loc=0x1505
fail_loc=0
PASS 1c (21s)

== sanity-lfsck test 2a: LFSCK can find out and repair crashed linkEA entry ========================================================== 15:04:41 (1787857481)
preparing... 1 * 1 files will be created Thu Aug 27 15:04:44 EDT 2026.
total: 1 mkdir in 0.01 seconds: 114.54 ops/second
total: 1 create in 0.01 seconds: 112.01 ops/second
total: 1 mkdir in 0.03 seconds: 30.86 ops/second
prepared Thu Aug 27 15:04:48 EDT 2026.
fail_loc=0x1603
fail_loc=0
192.168.204.120@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg420-client.virtnet /mnt/lustre (opts:)
Started LFSCK on the device lustre-MDT0000: scrub namespace
Starting client: oleg420-client.virtnet:  -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre
PASS 2a (20s)

== sanity-lfsck test 2b: LFSCK can find out and remove invalid linkEA entry ========================================================== 15:05:01 (1787857501)
preparing... 1 * 1 files will be created Thu Aug 27 15:05:05 EDT 2026.
total: 1 mkdir in 0.01 seconds: 144.09 ops/second
total: 1 create in 0.01 seconds: 185.15 ops/second
total: 1 mkdir in 0.00 seconds: 234.69 ops/second
prepared Thu Aug 27 15:05:09 EDT 2026.
fail_loc=0x1604
fail_loc=0
192.168.204.120@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg420-client.virtnet /mnt/lustre (opts:)
Started LFSCK on the device lustre-MDT0000: scrub namespace
Starting client: oleg420-client.virtnet:  -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre
PASS 2b (21s)

== sanity-lfsck test 2c: LFSCK can find out and remove repeated linkEA entry ========================================================== 15:05:22 (1787857522)
preparing... 1 * 1 files will be created Thu Aug 27 15:05:26 EDT 2026.
total: 1 mkdir in 0.01 seconds: 112.21 ops/second
total: 1 create in 0.01 seconds: 160.93 ops/second
total: 1 mkdir in 0.02 seconds: 57.75 ops/second
prepared Thu Aug 27 15:05:30 EDT 2026.
fail_loc=0x1605
fail_loc=0
192.168.204.120@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg420-client.virtnet /mnt/lustre (opts:)
Started LFSCK on the device lustre-MDT0000: scrub namespace
Starting client: oleg420-client.virtnet:  -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre
PASS 2c (20s)

== sanity-lfsck test 2d: LFSCK can recover the missing linkEA entry ========================================================== 15:05:42 (1787857542)
preparing... 1 * 1 files will be created Thu Aug 27 15:05:45 EDT 2026.
total: 1 mkdir in 0.01 seconds: 122.19 ops/second
total: 1 create in 0.01 seconds: 146.31 ops/second
total: 1 mkdir in 0.02 seconds: 61.09 ops/second
prepared Thu Aug 27 15:05:49 EDT 2026.
fail_loc=0x161d
fail_loc=0
192.168.204.120@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg420-client.virtnet /mnt/lustre (opts:)
Started LFSCK on the device lustre-MDT0000: scrub namespace
Starting client: oleg420-client.virtnet:  -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre
PASS 2d (20s)

== sanity-lfsck test 2e: namespace LFSCK can verify remote object linkEA ========================================================== 15:06:02 (1787857562)

 SKIP: sanity-lfsck test_2e needs >= 2 MDTs
SKIP 2e (3s)

== sanity-lfsck test 3: LFSCK can verify multiple-linked objects ========================================================== 15:06:05 (1787857565)
preparing... 4 * 4 files will be created Thu Aug 27 15:06:08 EDT 2026.
total: 4 mkdir in 0.04 seconds: 108.85 ops/second
total: 4 create in 0.03 seconds: 121.70 ops/second
total: 4 mkdir in 0.03 seconds: 140.76 ops/second
prepared Thu Aug 27 15:06:12 EDT 2026.
fail_loc=0x1603
fail_loc=0x1604
fail_loc=0
Started LFSCK on the device lustre-MDT0000: scrub namespace
PASS 3 (20s)

== sanity-lfsck test 4: FID-in-dirent can be rebuilt after MDT file-level backup/restore ========================================================== 15:06:25 (1787857585)

 SKIP: sanity-lfsck test_4 OI Scrub not implemented for ZFS
SKIP 4 (3s)

== sanity-lfsck test 5: LFSCK can handle IGIF object upgrading ========================================================== 15:06:28 (1787857588)

 SKIP: sanity-lfsck test_5 OI Scrub not implemented for ZFS
SKIP 5 (4s)

== sanity-lfsck test 6a: LFSCK resumes from last checkpoint (1) ========================================================== 15:06:32 (1787857592)
preparing... 5 * 5 files will be created Thu Aug 27 15:06:35 EDT 2026.
total: 5 mkdir in 0.04 seconds: 114.61 ops/second
total: 5 create in 0.05 seconds: 110.62 ops/second
total: 5 mkdir in 0.04 seconds: 141.26 ops/second
prepared Thu Aug 27 15:06:39 EDT 2026.
fail_val=1
fail_loc=0x1600
Started LFSCK on the device lustre-MDT0000: scrub namespace
fail_loc=0x80001608
fail_val=1
fail_loc=0x1600
Started LFSCK on the device lustre-MDT0000: scrub namespace
fail_loc=0
fail_val=0
PASS 6a (26s)

== sanity-lfsck test 6b: LFSCK resumes from last checkpoint (2) ========================================================== 15:06:58 (1787857618)
preparing... 5 * 5 files will be created Thu Aug 27 15:07:02 EDT 2026.
total: 5 mkdir in 0.04 seconds: 113.74 ops/second
total: 5 create in 0.03 seconds: 162.06 ops/second
total: 5 mkdir in 0.04 seconds: 135.48 ops/second
prepared Thu Aug 27 15:07:06 EDT 2026.
fail_val=1
fail_loc=0x1601
Started LFSCK on the device lustre-MDT0000: scrub namespace
fail_loc=0x80001609
fail_val=1
fail_loc=0x1601
Started LFSCK on the device lustre-MDT0000: scrub namespace
Additional debug for 6b
name: lfsck_namespace
magic: 0xa06249ff
version: 2
status: scanning-phase1
flags:
param:
last_completed_time: 1787857612
time_since_last_completed: 30 seconds
latest_start_time: 1787857639
time_since_latest_start: 3 seconds
last_checkpoint_time: 1787857635
time_since_last_checkpoint: 7 seconds
latest_start_position: 1911, [0x200000bd7:0xe5:0x0], 0x0
last_checkpoint_position: 1908, [0x200000bd7:0xe5:0x0], 0x0
first_failure_position: N/A, N/A, N/A
checked_phase1: 6
checked_phase2: 0
updated_phase1: 0
updated_phase2: 0
failed_phase1: 0
failed_phase2: 0
directories: 3
dirent_repaired: 0
linkea_repaired: 0
nlinks_repaired: 0
multiple_linked_checked: 0
multiple_linked_repaired: 0
unknown_inconsistency: 0
unmatched_pairs_repaired: 0
dangling_repaired: 0
multiple_referenced_repaired: 0
bad_file_type_repaired: 0
lost_dirent_repaired: 0
local_lost_found_scanned: 0
local_lost_found_moved: 0
local_lost_found_skipped: 0
local_lost_found_failed: 0
striped_dirs_scanned: 0
striped_dirs_repaired: 0
striped_dirs_failed: 0
striped_dirs_disabled: 0
striped_dirs_skipped: 0
striped_shards_scanned: 0
striped_shards_repaired: 0
striped_shards_failed: 0
striped_shards_skipped: 0
name_hash_repaired: 0
linkea_overflow_cleared: 0
agent_entries_repaired: 0
success_count: 10
run_time_phase1: 11 seconds
run_time_phase2: 0 seconds
average_speed_phase1: 0 items/sec
average_speed_phase2: N/A
average_speed_total: 0 items/sec
real_time_speed_phase1: 0 items/sec
real_time_speed_phase2: N/A
current_position: 1910, [0x200000bd7:0xe5:0x0], 0xb4505bce4e0000
fail_loc=0
fail_val=0
PASS 6b (32s)

== sanity-lfsck test 7a: non-stopped LFSCK should auto restarts after MDS remount (1) ========================================================== 15:07:30 (1787857650)
preparing... 5 * 5 files will be created Thu Aug 27 15:07:34 EDT 2026.
total: 5 mkdir in 0.04 seconds: 141.35 ops/second
total: 5 create in 0.05 seconds: 92.59 ops/second
total: 5 mkdir in 0.03 seconds: 147.08 ops/second
prepared Thu Aug 27 15:07:40 EDT 2026.
192.168.204.120@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg420-client.virtnet /mnt/lustre (opts:)
fail_val=1
fail_loc=0x1601
Started LFSCK on the device lustre-MDT0000: scrub namespace
stop mds1
fail_loc=0
fail_val=0
start mds1
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
PASS 7a (68s)

== sanity-lfsck test 7b: non-stopped LFSCK should auto restarts after MDS remount (2) ========================================================== 15:08:39 (1787857719)
Checking servers environments
Checking clients oleg420-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg420-server'
oleg420-server: oleg420-server.virtnet: executing load_modules_local
oleg420-server: Loading modules from /home/green/git/lustre-release/lustre
oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg420-server: detected 4 online CPUs by sysfs
oleg420-server: MODOPTS_LIBCFS=
oleg420-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg420-server: mount.lustre: according to /etc/mtab lustre-mdt1/mdt1 is already mounted on /mnt/lustre-mds1
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 17
Start of lustre-mdt1/mdt1 on mds1 failed 17
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg420-server: mount.lustre: according to /etc/mtab lustre-ost1/ost1 is already mounted on /mnt/lustre-ost1
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of lustre-ost1/ost1 on ost1 failed 17
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
oleg420-server: mount.lustre: according to /etc/mtab lustre-ost2/ost2 is already mounted on /mnt/lustre-ost2
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of lustre-ost2/ost2 on ost2 failed 17
Starting client: oleg420-client.virtnet:  -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre
Starting client oleg420-client.virtnet:  -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre
Started clients oleg420-client.virtnet: 
192.168.204.120@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff96f784767000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff96f784767000.idle_timeout=debug
disable quota as required
preparing... 2 * 2 files will be created Thu Aug 27 15:09:32 EDT 2026.
total: 2 mkdir in 0.01 seconds: 179.37 ops/second
total: 2 create in 0.01 seconds: 134.16 ops/second
total: 2 mkdir in 0.02 seconds: 80.44 ops/second
prepared Thu Aug 27 15:09:36 EDT 2026.
fail_loc=0x1604
fail_val=1
fail_loc=0x1602
Started LFSCK on the device lustre-MDT0000: scrub namespace
192.168.204.120@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg420-client.virtnet /mnt/lustre (opts:)
stop mds1
fail_loc=0
fail_val=0
start mds1
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
PASS 7b (90s)

== sanity-lfsck test 8: LFSCK state machine ============== 15:10:09 (1787857809)
formatall
oleg420-server: oleg420-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg420-server: oleg420-server.virtnet: executing load_modules_local
oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
setupall
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg420-server: oleg420-server.virtnet: executing load_modules_local
oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
Using TIMEOUT=20
preparing... 20 * 20 files will be created Thu Aug 27 15:13:01 EDT 2026.
total: 20 mkdir in 0.17 seconds: 117.34 ops/second
total: 20 create in 0.13 seconds: 148.56 ops/second
total: 20 mkdir in 0.16 seconds: 126.35 ops/second
prepared Thu Aug 27 15:13:09 EDT 2026.
fail_loc=0x1603
fail_loc=0x1604
192.168.204.120@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg420-client.virtnet /mnt/lustre (opts:)
fail_val=2
fail_loc=0x1601
Started LFSCK on the device lustre-MDT0000: scrub namespace
Stopped LFSCK on the device lustre-MDT0000.
Started LFSCK on the device lustre-MDT0000: scrub namespace
fail_loc=0x80001609
fail_loc=0x1600
Started LFSCK on the device lustre-MDT0000: scrub namespace
fail_loc=0x160a
stop mds1
fail_loc=0x160b
start mds1
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
fail_loc=0x1601
Started LFSCK on the device lustre-MDT0000: scrub namespace
stop mds1
fail_loc=0x160b
start mds1
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
stop mds1
start mds1 without resume LFSCK
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
fail_val=2
fail_loc=0x1602
Started LFSCK on the device lustre-MDT0000: scrub namespace
fail_loc=0
fail_val=0
PASS 8 (281s)

== sanity-lfsck test 9a: LFSCK speed control (1) ========= 15:14:50 (1787858090)
Checking servers environments
Checking clients oleg420-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg420-server'
oleg420-server: oleg420-server.virtnet: executing load_modules_local
oleg420-server: Loading modules from /home/green/git/lustre-release/lustre
oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg420-server: detected 4 online CPUs by sysfs
oleg420-server: MODOPTS_LIBCFS=
oleg420-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg420-server: mount.lustre: according to /etc/mtab lustre-mdt1/mdt1 is already mounted on /mnt/lustre-mds1
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 17
Start of lustre-mdt1/mdt1 on mds1 failed 17
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg420-server: mount.lustre: according to /etc/mtab lustre-ost1/ost1 is already mounted on /mnt/lustre-ost1
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of lustre-ost1/ost1 on ost1 failed 17
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
oleg420-server: mount.lustre: according to /etc/mtab lustre-ost2/ost2 is already mounted on /mnt/lustre-ost2
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 17
seq.cli-lustre-OST0001-super.width=65536
Start of lustre-ost2/ost2 on ost2 failed 17
Starting client: oleg420-client.virtnet:  -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre
Starting client oleg420-client.virtnet:  -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre
Started clients oleg420-client.virtnet: 
192.168.204.120@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff96f7903c0800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff96f7903c0800.idle_timeout=debug
disable quota as required
 - open/close 582 (time 1787858162.65 total 10.04 last 57.98)
 - open/close 1324 (time 1787858172.65 total 20.04 last 74.15)
 - open/close 2055 (time 1787858182.66 total 30.05 last 73.09)
 - open/close 2820 (time 1787858192.66 total 40.05 last 76.50)
 - open/close 3588 (time 1787858202.67 total 50.06 last 76.68)
 - open/close 4321 (time 1787858212.69 total 60.08 last 73.20)
 - open/close 4926 (time 1787858222.71 total 70.10 last 60.37)
total: 5000 open/close in 71.40 seconds: 70.03 ops/second
Started LFSCK on the device lustre-MDT0000: scrub layout
PASS 9a (171s)

== sanity-lfsck test 9b: LFSCK speed control (2) ========= 15:17:41 (1787858261)
preparing... 0 * 0 files will be created Thu Aug 27 15:18:29 EDT 2026.
prepared Thu Aug 27 15:18:34 EDT 2026.
Preparing another 50 * 50 files (with error) at Thu Aug 27 15:18:34 EDT 2026.
fail_loc=0x1604
total: 50 mkdir in 0.72 seconds: 69.83 ops/second
total: 50 create in 0.70 seconds: 70.96 ops/second
fail_loc=0x160c
Started LFSCK on the device lustre-MDT0000: scrub namespace
Waiting 25s for 'stopped'
Updated after 2s: want 'stopped' got 'stopped'
fail_loc=0
Prepared at Thu Aug 27 15:19:05 EDT 2026.
Started LFSCK on the device lustre-MDT0000: scrub namespace
Waiting 90s for 'completed'
Updated after 2s: want 'completed' got 'completed'
PASS 9b (119s)

== sanity-lfsck test 10: System is available during LFSCK scanning ========================================================== 15:19:40 (1787858380)

 SKIP: sanity-lfsck test_10 lookup(..)/linkea on ZFS issue
SKIP 10 (3s)

== sanity-lfsck test 11a: LFSCK can rebuild lost last_id ========================================================== 15:19:43 (1787858383)
total: 64 open/close in 0.96 seconds: 66.98 ops/second
stopall
remove LAST_ID on ost1: idx=0
removed '/mnt/lustre-ost1/O/0/LAST_ID'
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
fail_val=3
fail_loc=0x160e
trigger LFSCK for layout on ost1 to rebuild the LAST_ID(s)
Started LFSCK on the device lustre-OST0000: scrub layout
fail_val=0
fail_loc=0
Waiting 32s for 'completed'
Updated after 2s: want 'completed' got 'completed'
the LAST_ID(s) should have been rebuilt
PASS 11a (171s)

== sanity-lfsck test 11b: LFSCK can rebuild crashed last_id ========================================================== 15:22:34 (1787858554)
Checking servers environments
Checking clients oleg420-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg420-server'
oleg420-server: oleg420-server.virtnet: executing load_modules_local
oleg420-server: Loading modules from /home/green/git/lustre-release/lustre
oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg420-server: detected 4 online CPUs by sysfs
oleg420-server: MODOPTS_LIBCFS=
oleg420-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg420-server: mount.lustre: according to /etc/mtab lustre-ost1/ost1 is already mounted on /mnt/lustre-ost1
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 17
seq.cli-lustre-OST0000-super.width=65536
Start of lustre-ost1/ost1 on ost1 failed 17
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
Started lustre-OST0001
Starting client: oleg420-client.virtnet:  -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre
Starting client oleg420-client.virtnet:  -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre
Started clients oleg420-client.virtnet: 
192.168.204.120@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff96f7a044d800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff96f7a044d800.idle_timeout=debug
disable quota as required
set fail_loc=0x160d to skip the updating LAST_ID on-disk
fail_loc=0x160d
 - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 2785 prealloc_next_id: 2754 count: 32
total: 64 open/close in 1.03 seconds: 62.21 ops/second
192.168.204.120@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0
Stopping client oleg420-client.virtnet /mnt/lustre (opts:)
Stopping /mnt/lustre-ost1 (opts:) on oleg420-server
fail_loc=0x215
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
Started lustre-OST0000
the on-disk LAST_ID should be smaller than the expected one
trigger LFSCK for layout on ost1 to rebuild the on-disk LAST_ID
Started LFSCK on the device lustre-OST0000: scrub layout
Stopping /mnt/lustre-ost1 (opts:) on oleg420-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
Started lustre-OST0000
the on-disk LAST_ID should have been rebuilt
fail_loc=0
Stopping clients: oleg420-client.virtnet /mnt/lustre (opts:)
Stopping clients: oleg420-client.virtnet /mnt/lustre2 (opts:)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg420-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg420-server
PASS 11b (149s)

== sanity-lfsck test 12a: single command to trigger LFSCK on all devices ========================================================== 15:25:03 (1787858703)

 SKIP: sanity-lfsck test_12a needs >= 2 MDTs
SKIP 12a (4s)

== sanity-lfsck test 12b: auto detect Lustre device ====== 15:25:07 (1787858707)
Checking servers environments
Checking clients oleg420-client.virtnet environments
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg420-server'
oleg420-server: oleg420-server.virtnet: executing load_modules_local
oleg420-server: Loading modules from /home/green/git/lustre-release/lustre
oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg420-server: detected 4 online CPUs by sysfs
oleg420-server: MODOPTS_LIBCFS=
oleg420-server: Force libcfs to create 2 CPU partitions
Setup mgs, mdt, osts
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
Started lustre-MDT0000
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
Started lustre-OST0000
Start ost2: mount -t lustre -o localrecov  lustre-ost2/ost2 /mnt/lustre-ost2
seq.cli-lustre-OST0001-super.width=65536
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
Started lustre-OST0001
Starting client: oleg420-client.virtnet:  -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre
Starting client oleg420-client.virtnet:  -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre
Started clients oleg420-client.virtnet: 
192.168.204.120@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose)
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff96f784765800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff96f784765800.idle_timeout=debug
disable quota as required
Start LFSCK without '-M' specified.
Started LFSCK on the device lustre-MDT0000: scrub layout namespace
PASS 12b (91s)

== sanity-lfsck test 13: LFSCK can repair crashed lmm_oi ========================================================== 15:26:38 (1787858798)
#####
The lmm_oi in layout EA should be consistent with the MDT-object
FID; otherwise, the LFSCK should re-generate the lmm_oi from the
MDT-object FID.
#####
Inject failure stub to simulate bad lmm_oi
fail_loc=0x160f
total: 1 open/close in 0.03 seconds: 31.78 ops/second
fail_loc=0
Trigger layout LFSCK to find out the bad lmm_oi and fix them
Started LFSCK on the device lustre-MDT0000: scrub layout
PASS 13 (10s)

== sanity-lfsck test 14a: LFSCK can repair MDT-object with dangling LOV EA reference (1) ========================================================== 15:26:48 (1787858808)
#####
The OST-object referenced by the MDT-object should be there;
otherwise, the LFSCK should re-create the missing OST-object.
without '--delay-create-ostobj' option.
#####
Inject failure stub to simulate dangling referenced MDT-object
fail_loc=0x1610
 - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 2881 prealloc_next_id: 2851 count: 31
total: 47 open/close in 0.70 seconds: 67.17 ops/second
touch: setting times of '/mnt/lustre/d14a.sanity-lfsck/guard0': No such file or directory
touch: setting times of '/mnt/lustre/d14a.sanity-lfsck/guard1': No such file or directory
fail_loc=0
debug=-1
debug_mb=150
debug=-1
debug_mb=150
 - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 2945 prealloc_next_id: 2916 count: 30
total: 30 open/close in 0.86 seconds: 34.78 ops/second
'ls' should fail because of dangling referenced MDT-object
Trigger layout LFSCK to find out dangling reference
Started LFSCK on the device lustre-MDT0000: scrub layout
'stat' should fail because of not repair dangling by default
Trigger layout LFSCK to repair dangling reference
Started LFSCK on the device lustre-MDT0000: scrub layout
Waiting 32s for 'completed'
Updated after 3s: want 'completed' got 'completed'
'stat' should success after layout LFSCK repairing
debug_mb=21
debug_mb=21
stopall to cleanup object cache
setupall
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg420-server: oleg420-server.virtnet: executing load_modules_local
oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
Using TIMEOUT=20
PASS 14a (149s)

== sanity-lfsck test 14b: LFSCK can repair MDT-object with dangling LOV EA reference (2) ========================================================== 15:29:18 (1787858958)
#####
The OST-object referenced by the MDT-object should be there;
otherwise, the LFSCK should re-create the missing OST-object.
with '--delay-create-ostobj' option.
#####
Inject failure stub to simulate dangling referenced MDT-object
fail_loc=0x1610
 - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 3009 prealloc_next_id: 2978 count: 32
total: 63 open/close in 1.08 seconds: 58.20 ops/second
touch: setting times of '/mnt/lustre/d14b.sanity-lfsck/guard': No such file or directory
fail_loc=0
debug=-1
debug_mb=150
debug=-1
debug_mb=150
 - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 3073 prealloc_next_id: 3042 count: 32
total: 32 open/close in 0.93 seconds: 34.53 ops/second
'ls' should fail because of dangling referenced MDT-object
Trigger layout LFSCK to find out dangling reference
Started LFSCK on the device lustre-MDT0000: scrub layout
'stat' should fail because of not repair dangling by default
Trigger layout LFSCK to repair dangling reference
Started LFSCK on the device lustre-MDT0000: scrub layout
'stat' should success after layout LFSCK repairing
debug_mb=21
debug_mb=21
stopall to cleanup object cache
setupall
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg420-server: oleg420-server.virtnet: executing load_modules_local
oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
Using TIMEOUT=20
PASS 14b (125s)

== sanity-lfsck test 15a: LFSCK can repair unmatched MDT-object/OST-object pairs (1) ========================================================== 15:31:22 (1787859082)
#####
If the OST-object referenced by the MDT-object back points
to some non-exist MDT-object, then the LFSCK should repair
the OST-object to back point to the right MDT-object.
#####
Inject failure stub to make the OST-object to back point to
non-exist MDT-object.
fail_loc=0x1611
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676901 s, 15.5 MB/s
257+0 records in
257+0 records out
1052672 bytes (1.1 MB, 1.0 MiB) copied, 0.150252 s, 7.0 MB/s
fail_loc=0
Trigger layout LFSCK to find out unmatched pairs and fix them
Started LFSCK on the device lustre-MDT0000: scrub layout
PASS 15a (15s)

== sanity-lfsck test 15b: LFSCK can repair unmatched MDT-object/OST-object pairs (2) ========================================================== 15:31:38 (1787859098)
#####
If the OST-object referenced by the MDT-object back points
to other MDT-object that doesn't recognize the OST-object,
then the LFSCK should repair it to back point to the right
MDT-object (the first one).
#####
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530573 s, 19.8 MB/s
Inject failure stub to make the OST-object to back point to
other MDT-object
fail_loc=0x1612
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501779 s, 20.9 MB/s
2+0 records in
2+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.15386 s, 13.6 MB/s
fail_loc=0
Trigger layout LFSCK to find out unmatched pairs and fix them
Started LFSCK on the device lustre-MDT0000: scrub layout
PASS 15b (14s)

== sanity-lfsck test 15c: LFSCK can repair unmatched MDT-object/OST-object pairs (3) ========================================================== 15:31:52 (1787859112)

 SKIP: sanity-lfsck test_15c needs >= 2 MDTs
SKIP 15c (3s)

== sanity-lfsck test 15d: LFSCK don't crash upon dir migration failure ========================================================== 15:31:56 (1787859116)

 SKIP: sanity-lfsck test_15d needs >= 2 MDTs
SKIP 15d (4s)

== sanity-lfsck test 16: LFSCK can repair inconsistent MDT-object/OST-object owner ========================================================== 15:31:59 (1787859119)
#####
If the OST-object's owner information does not match the owner
information stored in the MDT-object, then the LFSCK trust the
MDT-object and update the OST-object's owner information.
#####
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319435 s, 32.8 MB/s
running as uid/gid/euid/egid 500/500/500/500, groups: 500
 [createmany] [-o] [/mnt/lustre/d16.sanity-lfsck/d1/o] [100]
total: 100 open/close in 2.17 seconds: 46.08 ops/second
Inject failure stub to skip OST-object owner changing
fail_loc=0x1613
fail_loc=0
Trigger layout LFSCK to find out inconsistent OST-object owner
and fix them
Started LFSCK on the device lustre-MDT0000: scrub layout
PASS 16 (15s)

== sanity-lfsck test 17: LFSCK can repair multiple references ========================================================== 15:32:14 (1787859134)
#####
If more than one MDT-objects reference the same OST-object,
and the OST-object only recognizes one MDT-object, then the
LFSCK should create new OST-objects for such non-recognized
MDT-objects.
#####
Inject failure stub to make two MDT-objects to refernce
the OST-object
fail_val=0
fail_loc=0x1614
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507516 s, 20.7 MB/s
total: 1 open/close in 0.02 seconds: 44.95 ops/second
fail_loc=0
fail_val=0
/mnt/lustre/d17.sanity-lfsck/f0 and /mnt/lustre/d17.sanity-lfsck/guard use the same OST-objects
/mnt/lustre/d17.sanity-lfsck/f1 and /mnt/lustre/d17.sanity-lfsck/guard use the same OST-objects
Trigger layout LFSCK to find out multiple refenced MDT-objects
and fix them
Started LFSCK on the device lustre-MDT0000: scrub layout
/mnt/lustre/d17.sanity-lfsck/f0 and /mnt/lustre/d17.sanity-lfsck/guard should use diff OST-objects
2+0 records in
2+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0847495 s, 24.7 MB/s
/mnt/lustre/d17.sanity-lfsck/f1 and /mnt/lustre/d17.sanity-lfsck/guard should use diff OST-objects
2+0 records in
2+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0410786 s, 51.1 MB/s
PASS 17 (15s)

== sanity-lfsck test 18a: Find out orphan OST-object and repair it (1) ========================================================== 15:32:29 (1787859149)
#####
The target MDT-object is there, but related stripe information
is lost or partly lost. The LFSCK should regenerate the missing
layout EA entries.
#####
2+0 records in
2+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0678385 s, 30.9 MB/s
[0x200003ab1:0x80:0x0]
/mnt/lustre/d18a.sanity-lfsck/a1/f1
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	          3217	        0xc91	   0x240000400

2+0 records in
2+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0725131 s, 28.9 MB/s
[0x200003ab1:0x81:0x0]
/mnt/lustre/d18a.sanity-lfsck/f3
  lcm_layout_gen:    3
  lcm_mirror_count:  1
  lcm_entry_count:   2
    lcme_id:             1
    lcme_mirror_id:      0
    lcme_flags:          init
    lcme_extent.e_start: 0
    lcme_extent.e_end:   1048576
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: 0
      lmm_objects:
      -   0: { l_ost_idx:   0, l_fid: [0x240000400:0xc92:0x0] }

    lcme_id:             2
    lcme_mirror_id:      0
    lcme_flags:          init
    lcme_extent.e_start: 1048576
    lcme_extent.e_end:   EOF
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: 1
      lmm_objects:
      -   0: { l_ost_idx:   1, l_fid: [0x280000400:0xae6:0x0] }

Inject failure, to make the MDT-object lost its layout EA
fail_loc=0x1615
fail_loc=0
The file size should be incorrect since layout EA is lost
Trigger layout LFSCK on all devices to find out orphan OST-object
Started LFSCK on the device lustre-MDT0000: scrub layout
[0x200003ab1:0x80:0x0]
/mnt/lustre/d18a.sanity-lfsck/a1/f1
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    1
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	          3217	        0xc91	   0x240000400

[0x200003ab1:0x81:0x0]
/mnt/lustre/d18a.sanity-lfsck/f3
  lcm_layout_gen:    1
  lcm_mirror_count:  1
  lcm_entry_count:   2
    lcme_id:             1
    lcme_mirror_id:      0
    lcme_flags:          init
    lcme_extent.e_start: 0
    lcme_extent.e_end:   1048576
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    1
      lmm_stripe_offset: 0
      lmm_objects:
      -   0: { l_ost_idx:   0, l_fid: [0x240000400:0xc92:0x0] }

    lcme_id:             2
    lcme_mirror_id:      0
    lcme_flags:          init
    lcme_extent.e_start: 1048576
    lcme_extent.e_end:   EOF
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    1
      lmm_stripe_offset: 1
      lmm_objects:
      -   0: { l_ost_idx:   1, l_fid: [0x280000400:0xae6:0x0] }

The file size should be correct after layout LFSCK scanning
PASS 18a (17s)

 SKIP: sanity-lfsck test_18b skipping excluded test 18b

== sanity-lfsck test 18c: Find out orphan OST-object and repair it (3) ========================================================== 15:32:48 (1787859168)
#####
The target MDT-object is lost, and the OST-object FID is missing.
The LFSCK should re-create the MDT-object with new FID under the 
directory .lustre/lost+found/MDTxxxx.
#####
Inject failure, to simulate the case of missing parent FID
fail_loc=0x1617
2+0 records in
2+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.127357 s, 16.5 MB/s
/mnt/lustre/d18c.sanity-lfsck/a1/f1
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	          3219	        0xc93	   0x240000400

2+0 records in
2+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.137499 s, 15.3 MB/s
/mnt/lustre/d18c.sanity-lfsck/f3
  lcm_layout_gen:    3
  lcm_mirror_count:  1
  lcm_entry_count:   2
    lcme_id:             1
    lcme_mirror_id:      0
    lcme_flags:          init
    lcme_extent.e_start: 0
    lcme_extent.e_end:   1048576
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: 0
      lmm_objects:
      -   0: { l_ost_idx:   0, l_fid: [0x240000400:0xc94:0x0] }

    lcme_id:             2
    lcme_mirror_id:      0
    lcme_flags:          init
    lcme_extent.e_start: 1048576
    lcme_extent.e_end:   EOF
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: 1
      lmm_objects:
      -   0: { l_ost_idx:   1, l_fid: [0x280000400:0xae7:0x0] }

fail_loc=0
Inject failure, to simulate the case of missing the MDT-object
fail_loc=0x1616
fail_loc=0
Trigger layout LFSCK on all devices to find out orphan OST-object
Started LFSCK on the device lustre-MDT0000: scrub layout
total 13
144115188109410307 dr-x------ 3 root root 6656 Dec 31  1969 .
144115205289279489 drwx------ 2 root root 6656 Aug 27 15:13 MDT0000
There should NOT be some stub under .lustre/lost+found/MDT0001/
There should be some stub under .lustre/lost+found/MDT0000/
PASS 18c (20s)

== sanity-lfsck test 18d: Find out orphan OST-object and repair it (4) ========================================================== 15:33:07 (1787859187)
#####
The target MDT-object layout EA is corrupted, but the right
OST-object is still alive as orphan. The layout LFSCK will
not create new OST-object to occupy such slot.
#####
[0x200003ab1:0x8c:0x0]
/mnt/lustre/d18d.sanity-lfsck/a1/f1
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	          3221	        0xc95	   0x240000400

[0x200003ab1:0x8d:0x0]
/mnt/lustre/d18d.sanity-lfsck/a1/f2
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	          3222	        0xc96	   0x240000400

[0x200003ab1:0x8e:0x0]
/mnt/lustre/d18d.sanity-lfsck/a1/f3
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	          3223	        0xc97	   0x240000400

[0x200003ab1:0x8f:0x0]
/mnt/lustre/d18d.sanity-lfsck/a1/f4
  lcm_layout_gen:    2
  lcm_mirror_count:  1
  lcm_entry_count:   2
    lcme_id:             1
    lcme_mirror_id:      0
    lcme_flags:          init
    lcme_extent.e_start: 0
    lcme_extent.e_end:   1048576
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: 0
      lmm_objects:
      -   0: { l_ost_idx:   0, l_fid: [0x240000400:0xc98:0x0] }

    lcme_id:             2
    lcme_mirror_id:      0
    lcme_flags:          0
    lcme_extent.e_start: 1048576
    lcme_extent.e_end:   EOF
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: -1

Inject failure to make /mnt/lustre/d18d.sanity-lfsck/a1/f1 and /mnt/lustre/d18d.sanity-lfsck/a1/f2
to reference the same OST-object (which is f1's OST-object).
Then drop /mnt/lustre/d18d.sanity-lfsck/a1/f1 and its OST-object, so f2 becomes
dangling reference case, but f2's old OST-object is there.
The failure also makes /mnt/lustre/d18d.sanity-lfsck/a1/f3 and /mnt/lustre/d18d.sanity-lfsck/a1/f4
to reference the same OST-object (which is f3's OST-object).
Then drop /mnt/lustre/d18d.sanity-lfsck/a1/f3 and its OST-object, so f4 becomes
dangling reference case, but f4's old OST-object is there.

fail_loc=0x1618
fail_loc=0
stopall to cleanup object cache
setupall
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg420-server: oleg420-server.virtnet: executing load_modules_local
oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
Using TIMEOUT=20
Trigger layout LFSCK on all devices to find out orphan OST-object
Started LFSCK on the device lustre-MDT0000: scrub layout
The file size should be correct after layout LFSCK scanning
The LFSCK should find back the original data.
foo
[0x200003ab1:0x8d:0x0]
/mnt/lustre/d18d.sanity-lfsck/a1/f2
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    1
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	          3222	        0xc96	   0x240000400

foo
[0x200003ab1:0x8f:0x0]
/mnt/lustre/d18d.sanity-lfsck/a1/f4
  lcm_layout_gen:    2
  lcm_mirror_count:  1
  lcm_entry_count:   2
    lcme_id:             1
    lcme_mirror_id:      0
    lcme_flags:          init
    lcme_extent.e_start: 0
    lcme_extent.e_end:   1048576
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    1
      lmm_stripe_offset: 0
      lmm_objects:
      -   0: { l_ost_idx:   0, l_fid: [0x240000400:0xc98:0x0] }

    lcme_id:             2
    lcme_mirror_id:      0
    lcme_flags:          0
    lcme_extent.e_start: 1048576
    lcme_extent.e_end:   EOF
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: -1

PASS 18d (134s)

== sanity-lfsck test 18e: Find out orphan OST-object and repair it (5) ========================================================== 15:35:21 (1787859321)
#####
The target MDT-object layout EA slot is occpuied by some new
created OST-object when repair dangling reference case. Such
conflict OST-object has been modified by others. To keep the
new data, the LFSCK will create a new file to refernece this
old orphan OST-object.
#####
[0x200004281:0x4:0x0]
/mnt/lustre/d18e.sanity-lfsck/a1/f1
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	          3266	        0xcc2	   0x240000400

[0x200004281:0x5:0x0]
/mnt/lustre/d18e.sanity-lfsck/a1/f2
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	          3267	        0xcc3	   0x240000400

[0x200004281:0x6:0x0]
/mnt/lustre/d18e.sanity-lfsck/a1/f3
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	          3268	        0xcc4	   0x240000400

[0x200004281:0x7:0x0]
/mnt/lustre/d18e.sanity-lfsck/a1/f4
  lcm_layout_gen:    2
  lcm_mirror_count:  1
  lcm_entry_count:   2
    lcme_id:             1
    lcme_mirror_id:      0
    lcme_flags:          init
    lcme_extent.e_start: 0
    lcme_extent.e_end:   1048576
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: 0
      lmm_objects:
      -   0: { l_ost_idx:   0, l_fid: [0x240000400:0xcc5:0x0] }

    lcme_id:             2
    lcme_mirror_id:      0
    lcme_flags:          0
    lcme_extent.e_start: 1048576
    lcme_extent.e_end:   EOF
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: -1

Inject failure to make /mnt/lustre/d18e.sanity-lfsck/a1/f1 and /mnt/lustre/d18e.sanity-lfsck/a1/f2
to reference the same OST-object (which is f1's OST-object).
Then drop /mnt/lustre/d18e.sanity-lfsck/a1/f1 and its OST-object, so f2 becomes
dangling reference case, but f2's old OST-object is there.
Also the failure makes /mnt/lustre/d18e.sanity-lfsck/a1/f3 and /mnt/lustre/d18e.sanity-lfsck/a1/f4
to reference the same OST-object (which is f3's OST-object).
Then drop /mnt/lustre/d18e.sanity-lfsck/a1/f3 and its OST-object, so f4 becomes
dangling reference case, but f4's old OST-object is there.

fail_loc=0x1618
fail_loc=0
stopall to cleanup object cache
setupall
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg420-server: oleg420-server.virtnet: executing load_modules_local
oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1
Using TIMEOUT=20
fail_val=10
fail_loc=0x1602
debug=-1
debug_mb=150
debug=-1
debug_mb=150
Trigger layout LFSCK on all devices to find out orphan OST-object
Started LFSCK on the device lustre-MDT0000: scrub layout
Write new data to f2/f4 to modify the new created OST-object.
fail_val=0
fail_loc=0
Waiting 120s for 'completed'
Updated after 7s: want 'completed' got 'completed'
debug_mb=21
debug_mb=21
There should be stub file under .lustre/lost+found/MDT0000/
The stub file should keep the original f2 or f4 data
foo
[0x200000402:0x6:0x0]
/mnt/lustre/.lustre/lost+found/MDT0000/[0x200000402:0x6:0x0]-[0x200004281:0x7:0x0]-0-C-0
  lcm_layout_gen:    3
  lcm_mirror_count:  1
  lcm_entry_count:   1
    lcme_id:             1
    lcme_mirror_id:      0
    lcme_flags:          init
    lcme_extent.e_start: 0
    lcme_extent.e_end:   1048576
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    1
      lmm_stripe_offset: 0
      lmm_objects:
      -   0: { l_ost_idx:   0, l_fid: [0x240000400:0xcc5:0x0] }

foo
[0x200000402:0x5:0x0]
/mnt/lustre/.lustre/lost+found/MDT0000/[0x200000402:0x5:0x0]-[0x200004281:0x5:0x0]-0-C-0
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    1
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	          3267	        0xcc3	   0x240000400

The f2/f4 should contains new data.
dummy
[0x200004281:0x5:0x0]
/mnt/lustre/d18e.sanity-lfsck/a1/f2
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	          3266	        0xcc2	   0x240000400

dummy
[0x200004281:0x7:0x0]
/mnt/lustre/d18e.sanity-lfsck/a1/f4
  lcm_layout_gen:    2
  lcm_mirror_count:  1
  lcm_entry_count:   2
    lcme_id:             1
    lcme_mirror_id:      0
    lcme_flags:          init
    lcme_extent.e_start: 0
    lcme_extent.e_end:   1048576
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: 0
      lmm_objects:
      -   0: { l_ost_idx:   0, l_fid: [0x240000400:0xcc4:0x0] }

    lcme_id:             2
    lcme_mirror_id:      0
    lcme_flags:          0
    lcme_extent.e_start: 1048576
    lcme_extent.e_end:   EOF
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: -1

PASS 18e (166s)

== sanity-lfsck test 18f: Skip the failed OST(s) when handle orphan OST-objects ========================================================== 15:38:07 (1787859487)
#####
The target MDT-object is lost. The LFSCK should re-create the
MDT-object under .lustre/lost+found/MDTxxxx. If some OST fail
to verify some OST-object(s) during the first stage-scanning,
the LFSCK should skip orphan OST-objects for such OST. Others
should not be affected.
#####
2+0 records in
2+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0786961 s, 26.6 MB/s
2+0 records in
2+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0678772 s, 30.9 MB/s
2+0 records in
2+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0630336 s, 33.3 MB/s
/mnt/lustre/d18f.sanity-lfsck/a1/f1
lmm_stripe_count:  1
lmm_stripe_size:   4194304
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	          3299	        0xce3	   0x240000400

/mnt/lustre/d18f.sanity-lfsck/a2/f2
lmm_stripe_count:  2
lmm_stripe_size:   1048576
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	          3300	        0xce4	   0x240000400
	     1	          2850	        0xb22	   0x280000400

Inject failure, to simulate the case of missing the MDT-object
fail_loc=0x1616
fail_loc=0
Inject failure, to simulate the OST0 fail to handle
MDT0 LFSCK request during the first-stage scanning.
fail_loc=0x161c
fail_val=0
Trigger layout LFSCK on all devices to find out orphan OST-object
Started LFSCK on the device lustre-MDT0000: scrub layout
fail_loc=0
fail_val=0
Trigger layout LFSCK on all devices again to cleanup
Started LFSCK on the device lustre-MDT0000: scrub layout
PASS 18f (25s)

== sanity-lfsck test 18g: Find out orphan OST-object and repair it (7) ========================================================== 15:38:32 (1787859512)
#####
The target MDT-object is lost, but related OI mapping is there
The LFSCK should recreate the lost MDT-object without affected
by the stale OI mapping.
#####
2+0 records in
2+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0939531 s, 22.3 MB/s
[0x200004a51:0xe:0x0]
/mnt/lustre/d18g.sanity-lfsck/a1/f1
lmm_stripe_count:  2
lmm_stripe_size:   1048576
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	          3301	        0xce5	   0x240000400
	     1	          2851	        0xb23	   0x280000400

Inject failure to simulate lost MDT-object but keep OI mapping
fail_loc=0x162e
fail_loc=0
Trigger layout LFSCK on all devices to find out orphan OST-object
Started LFSCK on the device lustre-MDT0000: scrub layout
Move the files from ./lustre/lost+found/MDTxxxx to namespace
[0x200004a51:0xe:0x0]
/mnt/lustre/d18g.sanity-lfsck/a1/f1
lmm_stripe_count:  2
lmm_stripe_size:   1048576
lmm_pattern:       raid0
lmm_layout_gen:    2
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	          3301	        0xce5	   0x240000400
	     1	          2851	        0xb23	   0x280000400

PASS 18g (15s)

== sanity-lfsck test 18h: LFSCK can repair crashed PFL extent range ========================================================== 15:38:47 (1787859527)
#####
The PFL extent crashed. During the first cycle LFSCK scanning,
the layout LFSCK will keep the bad PFL file(s) there without
scanning its OST-object(s). Then in the second stage scanning,
the OST will return related OST-object(s) to the MDT as orphan.
And then the LFSCK on the MDT can rebuild the PFL extent with
the 'orphan(s)' stripe information.
#####
0+1 records in
0+1 records out
319130 bytes (319 kB, 312 KiB) copied, 0.0190869 s, 16.7 MB/s
Inject failure stub to simulate bad PFL extent range
fail_loc=0x162f
fail_loc=0
dd: error writing '/mnt/lustre/d18h.sanity-lfsck/f0': No data available
1+0 records in
0+0 records out
0 bytes copied, 0.00705997 s, 0.0 kB/s
Trigger layout LFSCK to find out the bad lmm_oi and fix them
Started LFSCK on the device lustre-MDT0000: scrub layout
Data in /mnt/lustre/d18h.sanity-lfsck/f0 should not be broken
Write should succeed after LFSCK repairing the bad PFL range
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192269 s, 54.5 MB/s
PASS 18h (17s)

== sanity-lfsck test 18i: LFSCK can repair crashed PFL with compression ========================================================== 15:39:04 (1787859544)
WARNING: (-Z) option specified, however no layout defined (-E), compression feature will not be enabled.