-----============= acceptance-small: sanity-lfsck ============----- Thu Aug 27 10:43:29 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg133-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-lfsck.*ex': No such file or directory excepting tests: 18b 23b /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing load_modules_local oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: detected 4 online CPUs by sysfs oleg133-server: MODOPTS_LIBCFS= oleg133-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg133-server: Force libcfs to create 2 CPU partitions client=34683228 MDS=34683228 OSS=34683228 Stopping clients: oleg133-client.virtnet /mnt/lustre (opts:) Stopping client oleg133-client.virtnet /mnt/lustre opts: Stopping clients: oleg133-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg133-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg133-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg133-server unloading modules via unload_modules_local on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: modules unloaded. === sanity-lfsck: start setup 10:44:58 (1787841898) === Stopping clients: oleg133-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg133-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: oleg133-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing load_modules_local oleg133-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: detected 4 online CPUs by sysfs oleg133-server: MODOPTS_LIBCFS= oleg133-server: Force libcfs to create 2 CPU partitions oleg133-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg133-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg133-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 Checking servers environments Checking clients oleg133-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing load_modules_local oleg133-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: detected 4 online CPUs by sysfs oleg133-server: MODOPTS_LIBCFS= oleg133-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre Starting client oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre Started clients oleg133-client.virtnet: 192.168.201.133@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff97ee54c9c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff97ee54c9c800.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' Updated after 2s: want 'procname_uid' got 'procname_uid' disable quota as required === sanity-lfsck: finish setup 10:48:03 (1787842083) === == sanity-lfsck test 0: Control LFSCK manually =========== 10:48:07 (1787842087) preparing... 3 * 3 files will be created Thu Aug 27 10:48:09 EDT 2026. total: 3 mkdir in 0.03 seconds: 92.01 ops/second total: 3 create in 0.05 seconds: 59.41 ops/second total: 3 mkdir in 0.03 seconds: 88.04 ops/second prepared Thu Aug 27 10:48:15 EDT 2026. fail_val=3 fail_loc=0x1600 Started LFSCK on the device lustre-MDT0000: scrub namespace name: lfsck_namespace magic: 0xa06249ff version: 2 status: scanning-phase1 flags: param: last_completed_time: N/A time_since_last_completed: N/A latest_start_time: 1787842097 time_since_latest_start: 1 seconds last_checkpoint_time: N/A time_since_last_checkpoint: N/A latest_start_position: 184, N/A, N/A last_checkpoint_position: N/A, N/A, N/A first_failure_position: N/A, N/A, N/A checked_phase1: 0 checked_phase2: 0 updated_phase1: 0 updated_phase2: 0 failed_phase1: 0 failed_phase2: 0 directories: 0 dirent_repaired: 0 linkea_repaired: 0 nlinks_repaired: 0 multiple_linked_checked: 0 multiple_linked_repaired: 0 unknown_inconsistency: 0 unmatched_pairs_repaired: 0 dangling_repaired: 0 multiple_referenced_repaired: 0 bad_file_type_repaired: 0 lost_dirent_repaired: 0 local_lost_found_scanned: 0 local_lost_found_moved: 0 local_lost_found_skipped: 0 local_lost_found_failed: 0 striped_dirs_scanned: 0 striped_dirs_repaired: 0 striped_dirs_failed: 0 striped_dirs_disabled: 0 striped_dirs_skipped: 0 striped_shards_scanned: 0 striped_shards_repaired: 0 striped_shards_failed: 0 striped_shards_skipped: 0 name_hash_repaired: 0 linkea_overflow_cleared: 0 agent_entries_repaired: 0 success_count: 0 run_time_phase1: 1 seconds run_time_phase2: 0 seconds average_speed_phase1: 0 items/sec average_speed_phase2: N/A average_speed_total: 0 items/sec real_time_speed_phase1: 0 items/sec real_time_speed_phase2: N/A current_position: 183, N/A, N/A Stopped LFSCK on the device lustre-MDT0000. Started LFSCK on the device lustre-MDT0000: scrub namespace fail_loc=0 fail_val=0 Started LFSCK on the device lustre-MDT0000: scrub namespace stopall, should NOT crash LU-3649 Stopping clients: oleg133-client.virtnet /mnt/lustre (opts:) Stopping client oleg133-client.virtnet /mnt/lustre opts: Stopping clients: oleg133-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg133-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg133-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg133-server PASS 0 (57s) == sanity-lfsck test 1a: LFSCK can find out and repair crashed FID-in-dirent ========================================================== 10:49:05 (1787842145) Checking servers environments Checking clients oleg133-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing load_modules_local oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg133-server: detected 4 online CPUs by sysfs oleg133-server: MODOPTS_LIBCFS= oleg133-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre Starting client oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre Started clients oleg133-client.virtnet: 192.168.201.133@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff97ee53643000.idle_timeout=debug osc.lustre-OST0001-osc-ffff97ee53643000.idle_timeout=debug disable quota as required preparing... 1 * 1 files will be created Thu Aug 27 10:50:29 EDT 2026. total: 1 mkdir in 0.04 seconds: 25.90 ops/second total: 1 create in 0.01 seconds: 125.61 ops/second total: 1 mkdir in 0.01 seconds: 68.44 ops/second prepared Thu Aug 27 10:50:36 EDT 2026. fail_loc=0x1501 fail_loc=0 192.168.201.133@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg133-client.virtnet /mnt/lustre (opts:) Started LFSCK on the device lustre-MDT0000: scrub namespace Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre fail_loc=0x1505 fail_loc=0 PASS 1a (108s) == sanity-lfsck test 1b: LFSCK can find out and repair the missing FID-in-LMA ========================================================== 10:50:53 (1787842253) SKIP: sanity-lfsck test_1b OI Scrub not implemented for ZFS SKIP 1b (3s) == sanity-lfsck test 1c: LFSCK can find out and repair lost FID-in-dirent ========================================================== 10:50:56 (1787842256) preparing... 1 * 1 files will be created Thu Aug 27 10:51:00 EDT 2026. total: 1 mkdir in 0.01 seconds: 146.46 ops/second total: 1 create in 0.01 seconds: 161.69 ops/second total: 1 mkdir in 0.01 seconds: 195.24 ops/second prepared Thu Aug 27 10:51:04 EDT 2026. fail_loc=0x1504 fail_loc=0 192.168.201.133@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg133-client.virtnet /mnt/lustre (opts:) Started LFSCK on the device lustre-MDT0000: scrub namespace Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre fail_loc=0x1505 fail_loc=0 PASS 1c (22s) == sanity-lfsck test 2a: LFSCK can find out and repair crashed linkEA entry ========================================================== 10:51:18 (1787842278) preparing... 1 * 1 files will be created Thu Aug 27 10:51:21 EDT 2026. total: 1 mkdir in 0.01 seconds: 80.45 ops/second total: 1 create in 0.01 seconds: 77.36 ops/second total: 1 mkdir in 0.02 seconds: 43.44 ops/second prepared Thu Aug 27 10:51:26 EDT 2026. fail_loc=0x1603 fail_loc=0 192.168.201.133@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg133-client.virtnet /mnt/lustre (opts:) Started LFSCK on the device lustre-MDT0000: scrub namespace Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre PASS 2a (22s) == sanity-lfsck test 2b: LFSCK can find out and remove invalid linkEA entry ========================================================== 10:51:41 (1787842301) preparing... 1 * 1 files will be created Thu Aug 27 10:51:44 EDT 2026. total: 1 mkdir in 0.01 seconds: 178.44 ops/second total: 1 create in 0.01 seconds: 116.38 ops/second total: 1 mkdir in 0.01 seconds: 134.59 ops/second prepared Thu Aug 27 10:51:49 EDT 2026. fail_loc=0x1604 fail_loc=0 192.168.201.133@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg133-client.virtnet /mnt/lustre (opts:) Started LFSCK on the device lustre-MDT0000: scrub namespace Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre PASS 2b (22s) == sanity-lfsck test 2c: LFSCK can find out and remove repeated linkEA entry ========================================================== 10:52:02 (1787842322) preparing... 1 * 1 files will be created Thu Aug 27 10:52:06 EDT 2026. total: 1 mkdir in 0.01 seconds: 172.77 ops/second total: 1 create in 0.01 seconds: 174.43 ops/second total: 1 mkdir in 0.01 seconds: 107.96 ops/second prepared Thu Aug 27 10:52:11 EDT 2026. fail_loc=0x1605 fail_loc=0 192.168.201.133@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg133-client.virtnet /mnt/lustre (opts:) Started LFSCK on the device lustre-MDT0000: scrub namespace Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre PASS 2c (22s) == sanity-lfsck test 2d: LFSCK can recover the missing linkEA entry ========================================================== 10:52:25 (1787842345) preparing... 1 * 1 files will be created Thu Aug 27 10:52:28 EDT 2026. total: 1 mkdir in 0.01 seconds: 154.65 ops/second total: 1 create in 0.01 seconds: 134.91 ops/second total: 1 mkdir in 0.01 seconds: 66.98 ops/second prepared Thu Aug 27 10:52:34 EDT 2026. fail_loc=0x161d fail_loc=0 192.168.201.133@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg133-client.virtnet /mnt/lustre (opts:) Started LFSCK on the device lustre-MDT0000: scrub namespace Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre PASS 2d (24s) == sanity-lfsck test 2e: namespace LFSCK can verify remote object linkEA ========================================================== 10:52:49 (1787842369) SKIP: sanity-lfsck test_2e needs >= 2 MDTs SKIP 2e (4s) == sanity-lfsck test 3: LFSCK can verify multiple-linked objects ========================================================== 10:52:53 (1787842373) preparing... 4 * 4 files will be created Thu Aug 27 10:52:57 EDT 2026. total: 4 mkdir in 0.03 seconds: 130.82 ops/second total: 4 create in 0.04 seconds: 103.33 ops/second total: 4 mkdir in 0.03 seconds: 151.18 ops/second prepared Thu Aug 27 10:53:02 EDT 2026. fail_loc=0x1603 fail_loc=0x1604 fail_loc=0 Started LFSCK on the device lustre-MDT0000: scrub namespace PASS 3 (23s) == sanity-lfsck test 4: FID-in-dirent can be rebuilt after MDT file-level backup/restore ========================================================== 10:53:16 (1787842396) SKIP: sanity-lfsck test_4 OI Scrub not implemented for ZFS SKIP 4 (4s) == sanity-lfsck test 5: LFSCK can handle IGIF object upgrading ========================================================== 10:53:20 (1787842400) SKIP: sanity-lfsck test_5 OI Scrub not implemented for ZFS SKIP 5 (5s) == sanity-lfsck test 6a: LFSCK resumes from last checkpoint (1) ========================================================== 10:53:25 (1787842405) preparing... 5 * 5 files will be created Thu Aug 27 10:53:29 EDT 2026. total: 5 mkdir in 0.04 seconds: 131.98 ops/second total: 5 create in 0.03 seconds: 164.46 ops/second total: 5 mkdir in 0.05 seconds: 102.52 ops/second prepared Thu Aug 27 10:53:34 EDT 2026. fail_val=1 fail_loc=0x1600 Started LFSCK on the device lustre-MDT0000: scrub namespace fail_loc=0x80001608 fail_val=1 fail_loc=0x1600 Started LFSCK on the device lustre-MDT0000: scrub namespace fail_loc=0 fail_val=0 Waiting 32s for 'completed' Updated after 2s: want 'completed' got 'completed' PASS 6a (34s) == sanity-lfsck test 6b: LFSCK resumes from last checkpoint (2) ========================================================== 10:53:59 (1787842439) preparing... 5 * 5 files will be created Thu Aug 27 10:54:04 EDT 2026. total: 5 mkdir in 0.04 seconds: 129.30 ops/second total: 5 create in 0.03 seconds: 146.99 ops/second total: 5 mkdir in 0.04 seconds: 131.24 ops/second prepared Thu Aug 27 10:54:09 EDT 2026. fail_val=1 fail_loc=0x1601 Started LFSCK on the device lustre-MDT0000: scrub namespace fail_loc=0x80001609 fail_val=1 fail_loc=0x1601 Started LFSCK on the device lustre-MDT0000: scrub namespace Additional debug for 6b name: lfsck_namespace magic: 0xa06249ff version: 2 status: scanning-phase1 flags: param: last_completed_time: 1787842431 time_since_last_completed: 37 seconds latest_start_time: 1787842463 time_since_latest_start: 5 seconds last_checkpoint_time: 1787842458 time_since_last_checkpoint: 10 seconds latest_start_position: 2151, [0x200000bd7:0x133:0x0], 0x0 last_checkpoint_position: 2148, [0x200000bd7:0x133:0x0], 0x0 first_failure_position: N/A, N/A, N/A checked_phase1: 7 checked_phase2: 0 updated_phase1: 0 updated_phase2: 0 failed_phase1: 0 failed_phase2: 0 directories: 3 dirent_repaired: 0 linkea_repaired: 0 nlinks_repaired: 0 multiple_linked_checked: 0 multiple_linked_repaired: 0 unknown_inconsistency: 0 unmatched_pairs_repaired: 0 dangling_repaired: 0 multiple_referenced_repaired: 0 bad_file_type_repaired: 0 lost_dirent_repaired: 0 local_lost_found_scanned: 0 local_lost_found_moved: 0 local_lost_found_skipped: 0 local_lost_found_failed: 0 striped_dirs_scanned: 0 striped_dirs_repaired: 0 striped_dirs_failed: 0 striped_dirs_disabled: 0 striped_dirs_skipped: 0 striped_shards_scanned: 0 striped_shards_repaired: 0 striped_shards_failed: 0 striped_shards_skipped: 0 name_hash_repaired: 0 linkea_overflow_cleared: 0 agent_entries_repaired: 0 success_count: 10 run_time_phase1: 12 seconds run_time_phase2: 0 seconds average_speed_phase1: 0 items/sec average_speed_phase2: N/A average_speed_total: 0 items/sec real_time_speed_phase1: 0 items/sec real_time_speed_phase2: N/A current_position: 2150, [0x200000bd7:0x133:0x0], 0x306b8b43b95d8000 fail_loc=0 fail_val=0 PASS 6b (37s) == sanity-lfsck test 7a: non-stopped LFSCK should auto restarts after MDS remount (1) ========================================================== 10:54:36 (1787842476) preparing... 5 * 5 files will be created Thu Aug 27 10:54:41 EDT 2026. total: 5 mkdir in 0.06 seconds: 87.13 ops/second total: 5 create in 0.05 seconds: 99.05 ops/second total: 5 mkdir in 0.06 seconds: 81.65 ops/second prepared Thu Aug 27 10:54:47 EDT 2026. 192.168.201.133@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg133-client.virtnet /mnt/lustre (opts:) fail_val=1 fail_loc=0x1601 Started LFSCK on the device lustre-MDT0000: scrub namespace stop mds1 fail_loc=0 fail_val=0 start mds1 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 PASS 7a (69s) == sanity-lfsck test 7b: non-stopped LFSCK should auto restarts after MDS remount (2) ========================================================== 10:55:46 (1787842546) Checking servers environments Checking clients oleg133-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing load_modules_local oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg133-server: detected 4 online CPUs by sysfs oleg133-server: MODOPTS_LIBCFS= oleg133-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg133-server: mount.lustre: according to /etc/mtab lustre-mdt1/mdt1 is already mounted on /mnt/lustre-mds1 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 17 Start of lustre-mdt1/mdt1 on mds1 failed 17 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg133-server: mount.lustre: according to /etc/mtab lustre-ost1/ost1 is already mounted on /mnt/lustre-ost1 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 17 seq.cli-lustre-OST0000-super.width=65536 Start of lustre-ost1/ost1 on ost1 failed 17 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 oleg133-server: mount.lustre: according to /etc/mtab lustre-ost2/ost2 is already mounted on /mnt/lustre-ost2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 17 seq.cli-lustre-OST0001-super.width=65536 Start of lustre-ost2/ost2 on ost2 failed 17 Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre Starting client oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre Started clients oleg133-client.virtnet: 192.168.201.133@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff97ee511a4800.idle_timeout=debug osc.lustre-OST0001-osc-ffff97ee511a4800.idle_timeout=debug disable quota as required preparing... 2 * 2 files will be created Thu Aug 27 10:56:41 EDT 2026. total: 2 mkdir in 0.02 seconds: 96.31 ops/second total: 2 create in 0.04 seconds: 50.85 ops/second total: 2 mkdir in 0.02 seconds: 127.26 ops/second prepared Thu Aug 27 10:56:46 EDT 2026. fail_loc=0x1604 fail_val=1 fail_loc=0x1602 Started LFSCK on the device lustre-MDT0000: scrub namespace 192.168.201.133@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg133-client.virtnet /mnt/lustre (opts:) stop mds1 fail_loc=0 fail_val=0 start mds1 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 PASS 7b (94s) == sanity-lfsck test 8: LFSCK state machine ============== 10:57:19 (1787842639) formatall oleg133-server: oleg133-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg133-server: oleg133-server.virtnet: executing load_modules_local oleg133-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy setupall /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg133-server: oleg133-server.virtnet: executing load_modules_local oleg133-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Using TIMEOUT=20 preparing... 20 * 20 files will be created Thu Aug 27 10:59:58 EDT 2026. total: 20 mkdir in 0.23 seconds: 88.30 ops/second total: 20 create in 0.15 seconds: 133.62 ops/second total: 20 mkdir in 0.13 seconds: 151.15 ops/second prepared Thu Aug 27 11:00:07 EDT 2026. fail_loc=0x1603 fail_loc=0x1604 192.168.201.133@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg133-client.virtnet /mnt/lustre (opts:) fail_val=2 fail_loc=0x1601 Started LFSCK on the device lustre-MDT0000: scrub namespace Stopped LFSCK on the device lustre-MDT0000. Started LFSCK on the device lustre-MDT0000: scrub namespace fail_loc=0x80001609 fail_loc=0x1600 Started LFSCK on the device lustre-MDT0000: scrub namespace fail_loc=0x160a stop mds1 fail_loc=0x160b start mds1 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 fail_loc=0x1601 Started LFSCK on the device lustre-MDT0000: scrub namespace stop mds1 fail_loc=0x160b start mds1 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 stop mds1 start mds1 without resume LFSCK oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 fail_val=2 fail_loc=0x1602 Started LFSCK on the device lustre-MDT0000: scrub namespace fail_loc=0 fail_val=0 PASS 8 (270s) == sanity-lfsck test 9a: LFSCK speed control (1) ========= 11:01:49 (1787842909) Checking servers environments Checking clients oleg133-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing load_modules_local oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg133-server: detected 4 online CPUs by sysfs oleg133-server: MODOPTS_LIBCFS= oleg133-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg133-server: mount.lustre: according to /etc/mtab lustre-mdt1/mdt1 is already mounted on /mnt/lustre-mds1 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 17 Start of lustre-mdt1/mdt1 on mds1 failed 17 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg133-server: mount.lustre: according to /etc/mtab lustre-ost1/ost1 is already mounted on /mnt/lustre-ost1 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 17 seq.cli-lustre-OST0000-super.width=65536 Start of lustre-ost1/ost1 on ost1 failed 17 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 oleg133-server: mount.lustre: according to /etc/mtab lustre-ost2/ost2 is already mounted on /mnt/lustre-ost2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 17 seq.cli-lustre-OST0001-super.width=65536 Start of lustre-ost2/ost2 on ost2 failed 17 Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre Starting client oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre Started clients oleg133-client.virtnet: 192.168.201.133@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff97ee4a567000.idle_timeout=debug osc.lustre-OST0001-osc-ffff97ee4a567000.idle_timeout=debug disable quota as required - open/close 560 (time 1787842986.55 total 10.01 last 55.92) - open/close 1106 (time 1787842996.56 total 20.03 last 54.54) - open/close 1752 (time 1787843006.58 total 30.05 last 64.47) - open/close 2369 (time 1787843016.59 total 40.06 last 61.61) - open/close 3024 (time 1787843026.60 total 50.06 last 65.48) - open/close 3664 (time 1787843036.60 total 60.06 last 64.00) - open/close 4386 (time 1787843046.61 total 70.08 last 72.10) total: 5000 open/close in 77.98 seconds: 64.12 ops/second Started LFSCK on the device lustre-MDT0000: scrub layout PASS 9a (182s) == sanity-lfsck test 9b: LFSCK speed control (2) ========= 11:04:51 (1787843091) preparing... 0 * 0 files will be created Thu Aug 27 11:05:44 EDT 2026. prepared Thu Aug 27 11:05:48 EDT 2026. Preparing another 50 * 50 files (with error) at Thu Aug 27 11:05:48 EDT 2026. fail_loc=0x1604 total: 50 mkdir in 0.67 seconds: 74.40 ops/second total: 50 create in 0.51 seconds: 97.16 ops/second fail_loc=0x160c Started LFSCK on the device lustre-MDT0000: scrub namespace Waiting 25s for 'stopped' Updated after 2s: want 'stopped' got 'stopped' fail_loc=0 Prepared at Thu Aug 27 11:06:22 EDT 2026. Started LFSCK on the device lustre-MDT0000: scrub namespace PASS 9b (126s) == sanity-lfsck test 10: System is available during LFSCK scanning ========================================================== 11:06:57 (1787843217) SKIP: sanity-lfsck test_10 lookup(..)/linkea on ZFS issue SKIP 10 (4s) == sanity-lfsck test 11a: LFSCK can rebuild lost last_id ========================================================== 11:07:01 (1787843221) total: 64 open/close in 1.26 seconds: 50.66 ops/second stopall remove LAST_ID on ost1: idx=0 removed '/mnt/lustre-ost1/O/0/LAST_ID' oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 fail_val=3 fail_loc=0x160e trigger LFSCK for layout on ost1 to rebuild the LAST_ID(s) Started LFSCK on the device lustre-OST0000: scrub layout fail_val=0 fail_loc=0 Waiting 32s for 'completed' the LAST_ID(s) should have been rebuilt PASS 11a (163s) == sanity-lfsck test 11b: LFSCK can rebuild crashed last_id ========================================================== 11:09:44 (1787843384) Checking servers environments Checking clients oleg133-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing load_modules_local oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg133-server: detected 4 online CPUs by sysfs oleg133-server: MODOPTS_LIBCFS= oleg133-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg133-server: mount.lustre: according to /etc/mtab lustre-ost1/ost1 is already mounted on /mnt/lustre-ost1 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 17 seq.cli-lustre-OST0000-super.width=65536 Start of lustre-ost1/ost1 on ost1 failed 17 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre Starting client oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre Started clients oleg133-client.virtnet: 192.168.201.133@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff97ee504d0800.idle_timeout=debug osc.lustre-OST0001-osc-ffff97ee504d0800.idle_timeout=debug disable quota as required set fail_loc=0x160d to skip the updating LAST_ID on-disk fail_loc=0x160d - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 2785 prealloc_next_id: 2754 count: 32 total: 64 open/close in 1.53 seconds: 41.74 ops/second 192.168.201.133@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg133-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-ost1 (opts:) on oleg133-server fail_loc=0x215 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0000 the on-disk LAST_ID should be smaller than the expected one trigger LFSCK for layout on ost1 to rebuild the on-disk LAST_ID Started LFSCK on the device lustre-OST0000: scrub layout Stopping /mnt/lustre-ost1 (opts:) on oleg133-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0000 the on-disk LAST_ID should have been rebuilt fail_loc=0 Stopping clients: oleg133-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg133-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg133-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg133-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg133-server PASS 11b (158s) == sanity-lfsck test 12a: single command to trigger LFSCK on all devices ========================================================== 11:12:22 (1787843542) SKIP: sanity-lfsck test_12a needs >= 2 MDTs SKIP 12a (3s) == sanity-lfsck test 12b: auto detect Lustre device ====== 11:12:25 (1787843545) Checking servers environments Checking clients oleg133-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing load_modules_local oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg133-server: detected 4 online CPUs by sysfs oleg133-server: MODOPTS_LIBCFS= oleg133-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre Starting client oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre Started clients oleg133-client.virtnet: 192.168.201.133@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff97ee53541000.idle_timeout=debug osc.lustre-OST0001-osc-ffff97ee53541000.idle_timeout=debug disable quota as required Start LFSCK without '-M' specified. Started LFSCK on the device lustre-MDT0000: scrub layout namespace PASS 12b (89s) == sanity-lfsck test 13: LFSCK can repair crashed lmm_oi ========================================================== 11:13:55 (1787843635) ##### The lmm_oi in layout EA should be consistent with the MDT-object FID; otherwise, the LFSCK should re-generate the lmm_oi from the MDT-object FID. ##### Inject failure stub to simulate bad lmm_oi fail_loc=0x160f total: 1 open/close in 0.04 seconds: 23.89 ops/second fail_loc=0 Trigger layout LFSCK to find out the bad lmm_oi and fix them Started LFSCK on the device lustre-MDT0000: scrub layout PASS 13 (14s) == sanity-lfsck test 14a: LFSCK can repair MDT-object with dangling LOV EA reference (1) ========================================================== 11:14:08 (1787843648) ##### The OST-object referenced by the MDT-object should be there; otherwise, the LFSCK should re-create the missing OST-object. without '--delay-create-ostobj' option. ##### Inject failure stub to simulate dangling referenced MDT-object fail_loc=0x1610 - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 2881 prealloc_next_id: 2851 count: 31 total: 47 open/close in 0.97 seconds: 48.26 ops/second touch: setting times of '/mnt/lustre/d14a.sanity-lfsck/guard0': No such file or directory touch: setting times of '/mnt/lustre/d14a.sanity-lfsck/guard1': No such file or directory fail_loc=0 debug=-1 debug_mb=150 debug=-1 debug_mb=150 - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 2945 prealloc_next_id: 2916 count: 30 total: 30 open/close in 0.85 seconds: 35.47 ops/second 'ls' should fail because of dangling referenced MDT-object Trigger layout LFSCK to find out dangling reference Started LFSCK on the device lustre-MDT0000: scrub layout 'stat' should fail because of not repair dangling by default Trigger layout LFSCK to repair dangling reference Started LFSCK on the device lustre-MDT0000: scrub layout 'stat' should success after layout LFSCK repairing debug_mb=21 debug_mb=21 stopall to cleanup object cache setupall /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg133-server: oleg133-server.virtnet: executing load_modules_local oleg133-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Using TIMEOUT=20 PASS 14a (155s) == sanity-lfsck test 14b: LFSCK can repair MDT-object with dangling LOV EA reference (2) ========================================================== 11:16:44 (1787843804) ##### The OST-object referenced by the MDT-object should be there; otherwise, the LFSCK should re-create the missing OST-object. with '--delay-create-ostobj' option. ##### Inject failure stub to simulate dangling referenced MDT-object fail_loc=0x1610 - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 3009 prealloc_next_id: 2978 count: 32 total: 63 open/close in 1.33 seconds: 47.28 ops/second touch: setting times of '/mnt/lustre/d14b.sanity-lfsck/guard': No such file or directory fail_loc=0 debug=-1 debug_mb=150 debug=-1 debug_mb=150 - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 3073 prealloc_next_id: 3042 count: 32 total: 32 open/close in 0.81 seconds: 39.35 ops/second 'ls' should fail because of dangling referenced MDT-object Trigger layout LFSCK to find out dangling reference Started LFSCK on the device lustre-MDT0000: scrub layout 'stat' should fail because of not repair dangling by default Trigger layout LFSCK to repair dangling reference Started LFSCK on the device lustre-MDT0000: scrub layout 'stat' should success after layout LFSCK repairing debug_mb=21 debug_mb=21 stopall to cleanup object cache setupall /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg133-server: oleg133-server.virtnet: executing load_modules_local oleg133-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Using TIMEOUT=20 PASS 14b (129s) == sanity-lfsck test 15a: LFSCK can repair unmatched MDT-object/OST-object pairs (1) ========================================================== 11:18:53 (1787843933) ##### If the OST-object referenced by the MDT-object back points to some non-exist MDT-object, then the LFSCK should repair the OST-object to back point to the right MDT-object. ##### Inject failure stub to make the OST-object to back point to non-exist MDT-object. fail_loc=0x1611 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129008 s, 8.1 MB/s 257+0 records in 257+0 records out 1052672 bytes (1.1 MB, 1.0 MiB) copied, 0.19881 s, 5.3 MB/s fail_loc=0 Trigger layout LFSCK to find out unmatched pairs and fix them Started LFSCK on the device lustre-MDT0000: scrub layout PASS 15a (19s) == sanity-lfsck test 15b: LFSCK can repair unmatched MDT-object/OST-object pairs (2) ========================================================== 11:19:12 (1787843952) ##### If the OST-object referenced by the MDT-object back points to other MDT-object that doesn't recognize the OST-object, then the LFSCK should repair it to back point to the right MDT-object (the first one). ##### 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378098 s, 27.7 MB/s Inject failure stub to make the OST-object to back point to other MDT-object fail_loc=0x1612 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511284 s, 20.5 MB/s 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.114196 s, 18.4 MB/s fail_loc=0 Trigger layout LFSCK to find out unmatched pairs and fix them Started LFSCK on the device lustre-MDT0000: scrub layout PASS 15b (15s) == sanity-lfsck test 15c: LFSCK can repair unmatched MDT-object/OST-object pairs (3) ========================================================== 11:19:27 (1787843967) SKIP: sanity-lfsck test_15c needs >= 2 MDTs SKIP 15c (3s) == sanity-lfsck test 15d: LFSCK don't crash upon dir migration failure ========================================================== 11:19:30 (1787843970) SKIP: sanity-lfsck test_15d needs >= 2 MDTs SKIP 15d (4s) == sanity-lfsck test 16: LFSCK can repair inconsistent MDT-object/OST-object owner ========================================================== 11:19:34 (1787843974) ##### If the OST-object's owner information does not match the owner information stored in the MDT-object, then the LFSCK trust the MDT-object and update the OST-object's owner information. ##### 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424729 s, 24.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [createmany] [-o] [/mnt/lustre/d16.sanity-lfsck/d1/o] [100] total: 100 open/close in 1.73 seconds: 57.97 ops/second Inject failure stub to skip OST-object owner changing fail_loc=0x1613 fail_loc=0 Trigger layout LFSCK to find out inconsistent OST-object owner and fix them Started LFSCK on the device lustre-MDT0000: scrub layout PASS 16 (16s) == sanity-lfsck test 17: LFSCK can repair multiple references ========================================================== 11:19:51 (1787843991) ##### If more than one MDT-objects reference the same OST-object, and the OST-object only recognizes one MDT-object, then the LFSCK should create new OST-objects for such non-recognized MDT-objects. ##### Inject failure stub to make two MDT-objects to refernce the OST-object fail_val=0 fail_loc=0x1614 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610839 s, 17.2 MB/s total: 1 open/close in 0.03 seconds: 37.89 ops/second fail_loc=0 fail_val=0 /mnt/lustre/d17.sanity-lfsck/f0 and /mnt/lustre/d17.sanity-lfsck/guard use the same OST-objects /mnt/lustre/d17.sanity-lfsck/f1 and /mnt/lustre/d17.sanity-lfsck/guard use the same OST-objects Trigger layout LFSCK to find out multiple refenced MDT-objects and fix them Started LFSCK on the device lustre-MDT0000: scrub layout /mnt/lustre/d17.sanity-lfsck/f0 and /mnt/lustre/d17.sanity-lfsck/guard should use diff OST-objects 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0806572 s, 26.0 MB/s /mnt/lustre/d17.sanity-lfsck/f1 and /mnt/lustre/d17.sanity-lfsck/guard should use diff OST-objects 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.103241 s, 20.3 MB/s PASS 17 (15s) == sanity-lfsck test 18a: Find out orphan OST-object and repair it (1) ========================================================== 11:20:06 (1787844006) ##### The target MDT-object is there, but related stripe information is lost or partly lost. The LFSCK should regenerate the missing layout EA entries. ##### 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0836554 s, 25.1 MB/s [0x200003ab1:0x80:0x0] /mnt/lustre/d18a.sanity-lfsck/a1/f1 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3219 0xc93 0x240000400 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0719201 s, 29.2 MB/s [0x200003ab1:0x81:0x0] /mnt/lustre/d18a.sanity-lfsck/f3 lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc94:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xae4:0x0] } Inject failure, to make the MDT-object lost its layout EA fail_loc=0x1615 fail_loc=0 The file size should be incorrect since layout EA is lost Trigger layout LFSCK on all devices to find out orphan OST-object Started LFSCK on the device lustre-MDT0000: scrub layout [0x200003ab1:0x80:0x0] /mnt/lustre/d18a.sanity-lfsck/a1/f1 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 1 lmm_stripe_offset: 0 obdidx objid objid group 0 3219 0xc93 0x240000400 [0x200003ab1:0x81:0x0] /mnt/lustre/d18a.sanity-lfsck/f3 lcm_layout_gen: 1 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 1 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc94:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 1 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xae4:0x0] } The file size should be correct after layout LFSCK scanning PASS 18a (19s) SKIP: sanity-lfsck test_18b skipping excluded test 18b == sanity-lfsck test 18c: Find out orphan OST-object and repair it (3) ========================================================== 11:20:27 (1787844027) ##### The target MDT-object is lost, and the OST-object FID is missing. The LFSCK should re-create the MDT-object with new FID under the directory .lustre/lost+found/MDTxxxx. ##### Inject failure, to simulate the case of missing parent FID fail_loc=0x1617 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.154317 s, 13.6 MB/s /mnt/lustre/d18c.sanity-lfsck/a1/f1 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3221 0xc95 0x240000400 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.187107 s, 11.2 MB/s /mnt/lustre/d18c.sanity-lfsck/f3 lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc96:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xae5:0x0] } fail_loc=0 Inject failure, to simulate the case of missing the MDT-object fail_loc=0x1616 fail_loc=0 Trigger layout LFSCK on all devices to find out orphan OST-object Started LFSCK on the device lustre-MDT0000: scrub layout sanity-lfsck test_18c: @@@@@@ FAIL: (4) Expect 3 fixed on mds1, but got: 5 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7549:error() = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:2513:test_18c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7932:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7995:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7809:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:2540:main() Dumping lctl log to /tmp/testlogs//sanity-lfsck.test_18c.*.1787844045.log rsync: chown "/tmp/testlogs/.sanity-lfsck.test_18c.debug_log.oleg133-server.1787844045.log.wD8hAC" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lfsck.test_18c.dmesg.oleg133-server.1787844045.log.F6FNSU" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 18c (30s) == sanity-lfsck test 18d: Find out orphan OST-object and repair it (4) ========================================================== 11:21:05 (1787844065) ##### The target MDT-object layout EA is corrupted, but the right OST-object is still alive as orphan. The layout LFSCK will not create new OST-object to occupy such slot. ##### [0x200003ab1:0x8c:0x0] /mnt/lustre/d18d.sanity-lfsck/a1/f1 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3223 0xc97 0x240000400 [0x200003ab1:0x8d:0x0] /mnt/lustre/d18d.sanity-lfsck/a1/f2 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3224 0xc98 0x240000400 [0x200003ab1:0x8e:0x0] /mnt/lustre/d18d.sanity-lfsck/a1/f3 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3225 0xc99 0x240000400 [0x200003ab1:0x8f:0x0] /mnt/lustre/d18d.sanity-lfsck/a1/f4 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc9a:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 Inject failure to make /mnt/lustre/d18d.sanity-lfsck/a1/f1 and /mnt/lustre/d18d.sanity-lfsck/a1/f2 to reference the same OST-object (which is f1's OST-object). Then drop /mnt/lustre/d18d.sanity-lfsck/a1/f1 and its OST-object, so f2 becomes dangling reference case, but f2's old OST-object is there. The failure also makes /mnt/lustre/d18d.sanity-lfsck/a1/f3 and /mnt/lustre/d18d.sanity-lfsck/a1/f4 to reference the same OST-object (which is f3's OST-object). Then drop /mnt/lustre/d18d.sanity-lfsck/a1/f3 and its OST-object, so f4 becomes dangling reference case, but f4's old OST-object is there. fail_loc=0x1618 fail_loc=0 stopall to cleanup object cache setupall /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg133-server: oleg133-server.virtnet: executing load_modules_local oleg133-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Using TIMEOUT=20 Trigger layout LFSCK on all devices to find out orphan OST-object Started LFSCK on the device lustre-MDT0000: scrub layout The file size should be correct after layout LFSCK scanning The LFSCK should find back the original data. foo [0x200003ab1:0x8d:0x0] /mnt/lustre/d18d.sanity-lfsck/a1/f2 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 1 lmm_stripe_offset: 0 obdidx objid objid group 0 3224 0xc98 0x240000400 foo [0x200003ab1:0x8f:0x0] /mnt/lustre/d18d.sanity-lfsck/a1/f4 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 1 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc9a:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 18d (158s) == sanity-lfsck test 18e: Find out orphan OST-object and repair it (5) ========================================================== 11:23:43 (1787844223) ##### The target MDT-object layout EA slot is occpuied by some new created OST-object when repair dangling reference case. Such conflict OST-object has been modified by others. To keep the new data, the LFSCK will create a new file to refernece this old orphan OST-object. ##### [0x200004281:0x4:0x0] /mnt/lustre/d18e.sanity-lfsck/a1/f1 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3266 0xcc2 0x240000400 [0x200004281:0x5:0x0] /mnt/lustre/d18e.sanity-lfsck/a1/f2 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3267 0xcc3 0x240000400 [0x200004281:0x6:0x0] /mnt/lustre/d18e.sanity-lfsck/a1/f3 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3268 0xcc4 0x240000400 [0x200004281:0x7:0x0] /mnt/lustre/d18e.sanity-lfsck/a1/f4 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcc5:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 Inject failure to make /mnt/lustre/d18e.sanity-lfsck/a1/f1 and /mnt/lustre/d18e.sanity-lfsck/a1/f2 to reference the same OST-object (which is f1's OST-object). Then drop /mnt/lustre/d18e.sanity-lfsck/a1/f1 and its OST-object, so f2 becomes dangling reference case, but f2's old OST-object is there. Also the failure makes /mnt/lustre/d18e.sanity-lfsck/a1/f3 and /mnt/lustre/d18e.sanity-lfsck/a1/f4 to reference the same OST-object (which is f3's OST-object). Then drop /mnt/lustre/d18e.sanity-lfsck/a1/f3 and its OST-object, so f4 becomes dangling reference case, but f4's old OST-object is there. fail_loc=0x1618 fail_loc=0 stopall to cleanup object cache setupall /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg133-server: oleg133-server.virtnet: executing load_modules_local oleg133-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Using TIMEOUT=20 fail_val=10 fail_loc=0x1602 debug=-1 debug_mb=150 debug=-1 debug_mb=150 Trigger layout LFSCK on all devices to find out orphan OST-object Started LFSCK on the device lustre-MDT0000: scrub layout Write new data to f2/f4 to modify the new created OST-object. fail_val=0 fail_loc=0 Waiting 120s for 'completed' Updated after 6s: want 'completed' got 'completed' debug_mb=21 debug_mb=21 There should be stub file under .lustre/lost+found/MDT0000/ The stub file should keep the original f2 or f4 data foo [0x200000402:0x7:0x0] /mnt/lustre/.lustre/lost+found/MDT0000/[0x200000402:0x7:0x0]-[0x200004281:0x5:0x0]-0-C-0 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 1 lmm_stripe_offset: 0 obdidx objid objid group 0 3267 0xcc3 0x240000400 foo [0x200000402:0x8:0x0] /mnt/lustre/.lustre/lost+found/MDT0000/[0x200000402:0x8:0x0]-[0x200004281:0x7:0x0]-0-C-0 lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 1 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcc5:0x0] } The f2/f4 should contains new data. dummy [0x200004281:0x5:0x0] /mnt/lustre/d18e.sanity-lfsck/a1/f2 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3266 0xcc2 0x240000400 dummy [0x200004281:0x7:0x0] /mnt/lustre/d18e.sanity-lfsck/a1/f4 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcc4:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 18e (164s) == sanity-lfsck test 18f: Skip the failed OST(s) when handle orphan OST-objects ========================================================== 11:26:27 (1787844387) ##### The target MDT-object is lost. The LFSCK should re-create the MDT-object under .lustre/lost+found/MDTxxxx. If some OST fail to verify some OST-object(s) during the first stage-scanning, the LFSCK should skip orphan OST-objects for such OST. Others should not be affected. ##### 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0749226 s, 28.0 MB/s 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.1336 s, 15.7 MB/s 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.121419 s, 17.3 MB/s /mnt/lustre/d18f.sanity-lfsck/a1/f1 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3299 0xce3 0x240000400 /mnt/lustre/d18f.sanity-lfsck/a2/f2 lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3300 0xce4 0x240000400 1 2850 0xb22 0x280000400 Inject failure, to simulate the case of missing the MDT-object fail_loc=0x1616 fail_loc=0 Inject failure, to simulate the OST0 fail to handle MDT0 LFSCK request during the first-stage scanning. fail_loc=0x161c fail_val=0 Trigger layout LFSCK on all devices to find out orphan OST-object Started LFSCK on the device lustre-MDT0000: scrub layout fail_loc=0 fail_val=0 Trigger layout LFSCK on all devices again to cleanup Started LFSCK on the device lustre-MDT0000: scrub layout PASS 18f (27s) == sanity-lfsck test 18g: Find out orphan OST-object and repair it (7) ========================================================== 11:26:55 (1787844415) ##### The target MDT-object is lost, but related OI mapping is there The LFSCK should recreate the lost MDT-object without affected by the stale OI mapping. ##### 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0915351 s, 22.9 MB/s [0x200004a51:0xe:0x0] /mnt/lustre/d18g.sanity-lfsck/a1/f1 lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3301 0xce5 0x240000400 1 2851 0xb23 0x280000400 Inject failure to simulate lost MDT-object but keep OI mapping fail_loc=0x162e fail_loc=0 Trigger layout LFSCK on all devices to find out orphan OST-object Started LFSCK on the device lustre-MDT0000: scrub layout Move the files from ./lustre/lost+found/MDTxxxx to namespace [0x200004a51:0xe:0x0] /mnt/lustre/d18g.sanity-lfsck/a1/f1 lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 2 lmm_stripe_offset: 0 obdidx objid objid group 0 3301 0xce5 0x240000400 1 2851 0xb23 0x280000400 PASS 18g (17s) == sanity-lfsck test 18h: LFSCK can repair crashed PFL extent range ========================================================== 11:27:13 (1787844433) ##### The PFL extent crashed. During the first cycle LFSCK scanning, the layout LFSCK will keep the bad PFL file(s) there without scanning its OST-object(s). Then in the second stage scanning, the OST will return related OST-object(s) to the MDT as orphan. And then the LFSCK on the MDT can rebuild the PFL extent with the 'orphan(s)' stripe information. ##### 0+1 records in 0+1 records out 319248 bytes (319 kB, 312 KiB) copied, 0.0230658 s, 13.8 MB/s Inject failure stub to simulate bad PFL extent range fail_loc=0x162f fail_loc=0 dd: error writing '/mnt/lustre/d18h.sanity-lfsck/f0': No data available 1+0 records in 0+0 records out 0 bytes copied, 0.0122171 s, 0.0 kB/s Trigger layout LFSCK to find out the bad lmm_oi and fix them Started LFSCK on the device lustre-MDT0000: scrub layout Data in /mnt/lustre/d18h.sanity-lfsck/f0 should not be broken Write should succeed after LFSCK repairing the bad PFL range 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290848 s, 36.1 MB/s PASS 18h (21s) == sanity-lfsck test 18i: LFSCK can repair crashed PFL with compression ========================================================== 11:27:34 (1787844454) WARNING: (-Z) option specified, however no layout defined (-E), compression feature will not be enabled.