-----============= acceptance-small: sanity-lfsck ============----- Tue Sep 8 06:53:46 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg131-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-lfsck.*ex': No such file or directory excepting tests: 18b 23b Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: kmemleak disabled oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: MODOPTS_LIBCFS= oleg131-server: Force libcfs to create 2 CPU partitions oleg131-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt client=34683431 MDS=34683431 OSS=34683431 Stopping clients: oleg131-client.virtnet /mnt/lustre (opts:) Stopping client oleg131-client.virtnet /mnt/lustre opts: Stopping clients: oleg131-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg131-server unloading modules via unload_modules_local on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing unload_modules_local oleg131-server: kmemleak disabled modules unloaded. === sanity-lfsck: start setup 06:55:32 (1788864932) === Stopping clients: oleg131-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg131-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 oleg131-server: oleg131-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: kmemleak disabled oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: MODOPTS_LIBCFS= oleg131-server: Force libcfs to create 2 CPU partitions oleg131-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg131-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg131-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg131-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/vdc pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Format mds2: /dev/vdd pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Format ost1: /dev/vde pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Format ost2: /dev/vdf pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Checking servers environments Checking clients oleg131-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: kmemleak disabled oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: MODOPTS_LIBCFS= oleg131-server: Force libcfs to create 2 CPU partitions oleg131-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on /dev/vdc Started lustre-MDT0000 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on /dev/vdd Started lustre-MDT0001 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on /dev/vde Started lustre-OST0000 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on /dev/vdf Started lustre-OST0001 Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre Starting client oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre Started clients oleg131-client.virtnet: 192.168.201.131@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9ca543954800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9ca543954800.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' Updated after 2s: want 'procname_uid' got 'procname_uid' disable quota as required osd-ldiskfs.track_declares_assert=1 === sanity-lfsck: finish setup 06:59:11 (1788865151) === == sanity-lfsck test 0: Control LFSCK manually =========== 06:59:14 (1788865154) preparing... 3 * 3 files will be created Tue Sep 8 06:59:16 EDT 2026. total: 3 mkdir in 0.01 seconds: 212.13 ops/second total: 3 create in 0.01 seconds: 214.79 ops/second total: 3 mkdir in 0.02 seconds: 185.86 ops/second prepared Tue Sep 8 06:59:24 EDT 2026. fail_val=3 fail_loc=0x1600 Started LFSCK on the device lustre-MDT0000: scrub namespace name: lfsck_namespace magic: 0xa06249ff version: 2 status: scanning-phase1 flags: param: last_completed_time: N/A time_since_last_completed: N/A latest_start_time: 1788865165 time_since_latest_start: 1 seconds last_checkpoint_time: N/A time_since_last_checkpoint: N/A latest_start_position: 13, N/A, N/A last_checkpoint_position: N/A, N/A, N/A first_failure_position: N/A, N/A, N/A checked_phase1: 0 checked_phase2: 0 updated_phase1: 0 updated_phase2: 0 failed_phase1: 0 failed_phase2: 0 directories: 0 dirent_repaired: 0 linkea_repaired: 0 nlinks_repaired: 0 multiple_linked_checked: 0 multiple_linked_repaired: 0 unknown_inconsistency: 0 unmatched_pairs_repaired: 0 dangling_repaired: 0 multiple_referenced_repaired: 0 bad_file_type_repaired: 0 lost_dirent_repaired: 0 local_lost_found_scanned: 0 local_lost_found_moved: 0 local_lost_found_skipped: 0 local_lost_found_failed: 0 striped_dirs_scanned: 0 striped_dirs_repaired: 0 striped_dirs_failed: 0 striped_dirs_disabled: 0 striped_dirs_skipped: 0 striped_shards_scanned: 0 striped_shards_repaired: 0 striped_shards_failed: 0 striped_shards_skipped: 0 name_hash_repaired: 0 linkea_overflow_cleared: 0 agent_entries_repaired: 0 success_count: 0 run_time_phase1: 1 seconds run_time_phase2: 0 seconds average_speed_phase1: 0 items/sec average_speed_phase2: N/A average_speed_total: 0 items/sec real_time_speed_phase1: 0 items/sec real_time_speed_phase2: N/A current_position: 12, N/A, N/A Stopped LFSCK on the device lustre-MDT0000. Started LFSCK on the device lustre-MDT0000: scrub namespace fail_loc=0 fail_val=0 Started LFSCK on the device lustre-MDT0000: scrub namespace stopall, should NOT crash LU-3649 Stopping clients: oleg131-client.virtnet /mnt/lustre (opts:) Stopping client oleg131-client.virtnet /mnt/lustre opts: Stopping clients: oleg131-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg131-server PASS 0 (78s) == sanity-lfsck test 1a: LFSCK can find out and repair crashed FID-in-dirent ========================================================== 07:00:32 (1788865232) Checking servers environments Checking clients oleg131-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: kmemleak disabled oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: MODOPTS_LIBCFS= oleg131-server: Force libcfs to create 2 CPU partitions oleg131-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0000 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0001 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre Starting client oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre Started clients oleg131-client.virtnet: 192.168.201.131@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9ca545bc8000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9ca545bc8000.idle_timeout=debug disable quota as required preparing... 1 * 1 files will be created Tue Sep 8 07:02:00 EDT 2026. total: 1 mkdir in 0.00 seconds: 258.60 ops/second total: 1 create in 0.01 seconds: 88.22 ops/second total: 1 mkdir in 0.01 seconds: 146.22 ops/second prepared Tue Sep 8 07:02:05 EDT 2026. fail_loc=0x1501 fail_loc=0 192.168.201.131@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg131-client.virtnet /mnt/lustre (opts:) Started LFSCK on the device lustre-MDT0000: scrub namespace oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre fail_loc=0x1505 fail_loc=0 PASS 1a (128s) == sanity-lfsck test 1b: LFSCK can find out and repair the missing FID-in-LMA ========================================================== 07:02:40 (1788865360) preparing... 1 * 1 files will be created Tue Sep 8 07:02:45 EDT 2026. total: 1 mkdir in 0.01 seconds: 138.70 ops/second total: 1 create in 0.00 seconds: 236.51 ops/second total: 1 mkdir in 0.01 seconds: 110.37 ops/second prepared Tue Sep 8 07:02:49 EDT 2026. fail_loc=0x1502 fail_loc=0 192.168.201.131@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg131-client.virtnet /mnt/lustre (opts:) fail_loc=0x1506 Started LFSCK on the device lustre-MDT0000: scrub namespace fail_loc=0 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre fail_loc=0x1505 fail_loc=0 PASS 1b (44s) == sanity-lfsck test 1c: LFSCK can find out and repair lost FID-in-dirent ========================================================== 07:03:24 (1788865404) preparing... 1 * 1 files will be created Tue Sep 8 07:03:27 EDT 2026. total: 1 mkdir in 0.01 seconds: 124.89 ops/second total: 1 create in 0.01 seconds: 142.37 ops/second total: 1 mkdir in 0.01 seconds: 114.69 ops/second prepared Tue Sep 8 07:03:32 EDT 2026. fail_loc=0x1504 fail_loc=0 192.168.201.131@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg131-client.virtnet /mnt/lustre (opts:) Started LFSCK on the device lustre-MDT0000: scrub namespace oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre fail_loc=0x1505 fail_loc=0 PASS 1c (50s) == sanity-lfsck test 2a: LFSCK can find out and repair crashed linkEA entry ========================================================== 07:04:14 (1788865454) preparing... 1 * 1 files will be created Tue Sep 8 07:04:17 EDT 2026. total: 1 mkdir in 0.00 seconds: 241.90 ops/second total: 1 create in 0.00 seconds: 254.06 ops/second total: 1 mkdir in 0.01 seconds: 185.42 ops/second prepared Tue Sep 8 07:04:21 EDT 2026. fail_loc=0x1603 fail_loc=0 192.168.201.131@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg131-client.virtnet /mnt/lustre (opts:) Started LFSCK on the device lustre-MDT0000: scrub namespace oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre PASS 2a (38s) == sanity-lfsck test 2b: LFSCK can find out and remove invalid linkEA entry ========================================================== 07:04:52 (1788865492) preparing... 1 * 1 files will be created Tue Sep 8 07:04:55 EDT 2026. total: 1 mkdir in 0.01 seconds: 89.16 ops/second total: 1 create in 0.01 seconds: 167.59 ops/second total: 1 mkdir in 0.01 seconds: 162.10 ops/second prepared Tue Sep 8 07:04:58 EDT 2026. fail_loc=0x1604 fail_loc=0 192.168.201.131@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg131-client.virtnet /mnt/lustre (opts:) Started LFSCK on the device lustre-MDT0000: scrub namespace oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre PASS 2b (42s) == sanity-lfsck test 2c: LFSCK can find out and remove repeated linkEA entry ========================================================== 07:05:34 (1788865534) preparing... 1 * 1 files will be created Tue Sep 8 07:05:38 EDT 2026. total: 1 mkdir in 0.03 seconds: 35.88 ops/second total: 1 create in 0.03 seconds: 29.91 ops/second total: 1 mkdir in 0.01 seconds: 73.99 ops/second prepared Tue Sep 8 07:05:43 EDT 2026. fail_loc=0x1605 fail_loc=0 192.168.201.131@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg131-client.virtnet /mnt/lustre (opts:) Started LFSCK on the device lustre-MDT0000: scrub namespace oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre PASS 2c (45s) == sanity-lfsck test 2d: LFSCK can recover the missing linkEA entry ========================================================== 07:06:19 (1788865579) preparing... 1 * 1 files will be created Tue Sep 8 07:06:23 EDT 2026. total: 1 mkdir in 0.01 seconds: 171.26 ops/second total: 1 create in 0.01 seconds: 125.74 ops/second total: 1 mkdir in 0.01 seconds: 112.07 ops/second prepared Tue Sep 8 07:06:27 EDT 2026. fail_loc=0x161d fail_loc=0 192.168.201.131@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg131-client.virtnet /mnt/lustre (opts:) Started LFSCK on the device lustre-MDT0000: scrub namespace oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre PASS 2d (44s) == sanity-lfsck test 2e: namespace LFSCK can verify remote object linkEA ========================================================== 07:07:03 (1788865623) fail_loc=0x1603 fail_loc=0 Started LFSCK on the device lustre-MDT0000: scrub namespace PASS 2e (12s) == sanity-lfsck test 3: LFSCK can verify multiple-linked objects ========================================================== 07:07:15 (1788865635) preparing... 4 * 4 files will be created Tue Sep 8 07:07:17 EDT 2026. total: 4 mkdir in 0.04 seconds: 108.37 ops/second total: 4 create in 0.02 seconds: 189.84 ops/second total: 4 mkdir in 0.06 seconds: 70.41 ops/second prepared Tue Sep 8 07:07:21 EDT 2026. fail_loc=0x1603 fail_loc=0x1604 fail_loc=0 Started LFSCK on the device lustre-MDT0000: scrub namespace PASS 3 (16s) == sanity-lfsck test 4: FID-in-dirent can be rebuilt after MDT file-level backup/restore ========================================================== 07:07:31 (1788865651) preparing... 3 * 3 files will be created Tue Sep 8 07:07:34 EDT 2026. total: 3 mkdir in 0.02 seconds: 178.80 ops/second total: 3 create in 0.02 seconds: 199.46 ops/second total: 3 mkdir in 0.01 seconds: 218.35 ops/second prepared Tue Sep 8 07:07:38 EDT 2026. Stopping clients: oleg131-client.virtnet /mnt/lustre (opts:) Stopping client oleg131-client.virtnet /mnt/lustre opts: file-level backup/restore on mds1:/dev/mapper/mds1_flakey backup data reformat new device Format mds1: /dev/mapper/mds1_flakey restore data remove recovery logs removed '/mnt/lustre-brpt/CATALOGS' start mds1 with disabling OI scrub oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 fail_val=1 fail_loc=0x1601 Started LFSCK on the device lustre-MDT0000: scrub namespace Waiting 32s for 'inconsistent' Updated after 2s: want 'inconsistent' got 'inconsistent' fail_loc=0 fail_val=0 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre fail_loc=0x1505 fail_loc=0 PASS 4 (130s) == sanity-lfsck test 5: LFSCK can handle IGIF object upgrading ========================================================== 07:09:41 (1788865781) preparing... 1 * 1 files will be created Tue Sep 8 07:09:44 EDT 2026. fail_loc=0x1504 total: 1 mkdir in 0.01 seconds: 160.82 ops/second total: 1 create in 0.00 seconds: 222.72 ops/second total: 1 mkdir in 0.01 seconds: 148.04 ops/second fail_loc=0 prepared Tue Sep 8 07:09:49 EDT 2026. Stopping clients: oleg131-client.virtnet /mnt/lustre (opts:) Stopping client oleg131-client.virtnet /mnt/lustre opts: file-level backup/restore on mds1:/dev/mapper/mds1_flakey backup data reformat new device Format mds1: /dev/mapper/mds1_flakey restore data remove recovery logs removed '/mnt/lustre-brpt/CATALOGS' start mds1 with disabling OI scrub oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 fail_val=1 fail_loc=0x1601 Started LFSCK on the device lustre-MDT0000: scrub namespace Waiting 32s for 'inconsistent,upgrade' Updated after 6s: want 'inconsistent,upgrade' got 'inconsistent,upgrade' fail_loc=0 fail_val=0 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre fail_loc=0x1505 fail_loc=0 PASS 5 (91s) == sanity-lfsck test 6a: LFSCK resumes from last checkpoint (1) ========================================================== 07:11:12 (1788865872) preparing... 5 * 5 files will be created Tue Sep 8 07:11:15 EDT 2026. total: 5 mkdir in 0.04 seconds: 125.47 ops/second total: 5 create in 0.05 seconds: 102.95 ops/second total: 5 mkdir in 0.03 seconds: 172.30 ops/second prepared Tue Sep 8 07:11:18 EDT 2026. fail_val=1 fail_loc=0x1600 Started LFSCK on the device lustre-MDT0000: scrub namespace fail_loc=0x80001608 fail_val=1 fail_loc=0x1600 Started LFSCK on the device lustre-MDT0000: scrub namespace fail_loc=0 fail_val=0 PASS 6a (25s) == sanity-lfsck test 6b: LFSCK resumes from last checkpoint (2) ========================================================== 07:11:37 (1788865897) preparing... 5 * 5 files will be created Tue Sep 8 07:11:41 EDT 2026. total: 5 mkdir in 0.09 seconds: 53.87 ops/second total: 5 create in 0.03 seconds: 169.03 ops/second total: 5 mkdir in 0.03 seconds: 171.12 ops/second prepared Tue Sep 8 07:11:44 EDT 2026. fail_val=1 fail_loc=0x1601 Started LFSCK on the device lustre-MDT0000: scrub namespace fail_loc=0x80001609 fail_val=1 fail_loc=0x1601 Started LFSCK on the device lustre-MDT0000: scrub namespace Additional debug for 6b name: lfsck_namespace magic: 0xa06249ff version: 2 status: scanning-phase1 flags: param: last_completed_time: 1788865890 time_since_last_completed: 30 seconds latest_start_time: 1788865917 time_since_latest_start: 3 seconds last_checkpoint_time: 1788865913 time_since_last_checkpoint: 7 seconds latest_start_position: 20113, [0x2000061c1:0xc5:0x0], 0x241c8e9b63f73066 last_checkpoint_position: 20111, [0x2000061c1:0xc5:0x0], 0x23fd9730d61e61cd first_failure_position: N/A, N/A, N/A checked_phase1: 7 checked_phase2: 0 updated_phase1: 0 updated_phase2: 0 failed_phase1: 0 failed_phase2: 0 directories: 2 dirent_repaired: 0 linkea_repaired: 0 nlinks_repaired: 0 multiple_linked_checked: 0 multiple_linked_repaired: 0 unknown_inconsistency: 0 unmatched_pairs_repaired: 0 dangling_repaired: 0 multiple_referenced_repaired: 0 bad_file_type_repaired: 0 lost_dirent_repaired: 0 local_lost_found_scanned: 0 local_lost_found_moved: 0 local_lost_found_skipped: 0 local_lost_found_failed: 0 striped_dirs_scanned: 0 striped_dirs_repaired: 0 striped_dirs_failed: 0 striped_dirs_disabled: 0 striped_dirs_skipped: 0 striped_shards_scanned: 0 striped_shards_repaired: 0 striped_shards_failed: 0 striped_shards_skipped: 0 name_hash_repaired: 0 linkea_overflow_cleared: 0 agent_entries_repaired: 0 success_count: 14 run_time_phase1: 11 seconds run_time_phase2: 0 seconds average_speed_phase1: 0 items/sec average_speed_phase2: N/A average_speed_total: 0 items/sec real_time_speed_phase1: 0 items/sec real_time_speed_phase2: N/A current_position: 20112, [0x2000061c1:0xc5:0x0], 0x4b975613a10e8ca0 fail_loc=0 fail_val=0 PASS 6b (31s) == sanity-lfsck test 7a: non-stopped LFSCK should auto restarts after MDS remount (1) ========================================================== 07:12:08 (1788865928) preparing... 5 * 5 files will be created Tue Sep 8 07:12:12 EDT 2026. total: 5 mkdir in 0.03 seconds: 149.80 ops/second total: 5 create in 0.02 seconds: 200.17 ops/second total: 5 mkdir in 0.04 seconds: 139.07 ops/second prepared Tue Sep 8 07:12:17 EDT 2026. 192.168.201.131@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg131-client.virtnet /mnt/lustre (opts:) fail_val=1 fail_loc=0x1601 Started LFSCK on the device lustre-MDT0000: scrub namespace stop mds1 fail_loc=0 fail_val=0 start mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 PASS 7a (45s) == sanity-lfsck test 7b: non-stopped LFSCK should auto restarts after MDS remount (2) ========================================================== 07:12:53 (1788865973) Checking servers environments Checking clients oleg131-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: kmemleak disabled oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: MODOPTS_LIBCFS= oleg131-server: Force libcfs to create 2 CPU partitions oleg131-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg131-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 17 Start of /dev/mapper/mds2_flakey on mds2 failed 17 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg131-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 17 seq.cli-lustre-OST0000-super.width=65536 Start of /dev/mapper/ost1_flakey on ost1 failed 17 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 oleg131-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 17 seq.cli-lustre-OST0001-super.width=65536 Start of /dev/mapper/ost2_flakey on ost2 failed 17 Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre Starting client oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre Started clients oleg131-client.virtnet: 192.168.201.131@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9ca543954800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9ca543954800.idle_timeout=debug disable quota as required preparing... 2 * 2 files will be created Tue Sep 8 07:14:03 EDT 2026. total: 2 mkdir in 0.01 seconds: 267.78 ops/second total: 2 create in 0.02 seconds: 130.51 ops/second total: 2 mkdir in 0.02 seconds: 111.06 ops/second prepared Tue Sep 8 07:14:08 EDT 2026. fail_loc=0x1604 fail_val=1 fail_loc=0x1602 Started LFSCK on the device lustre-MDT0000: scrub namespace 192.168.201.131@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg131-client.virtnet /mnt/lustre (opts:) stop mds1 fail_loc=0 fail_val=0 start mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 PASS 7b (111s) == sanity-lfsck test 8: LFSCK state machine ============== 07:14:44 (1788866084) formatall oleg131-server: oleg131-server.virtnet: executing set_hostid oleg131-server: oleg131-server.virtnet: executing load_modules_local setupall oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Using TIMEOUT=20 preparing... 20 * 20 files will be created Tue Sep 8 07:17:55 EDT 2026. total: 20 mkdir in 0.13 seconds: 149.18 ops/second total: 20 create in 0.20 seconds: 101.89 ops/second total: 20 mkdir in 0.17 seconds: 117.96 ops/second prepared Tue Sep 8 07:18:04 EDT 2026. fail_loc=0x1603 fail_loc=0x1604 192.168.201.131@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg131-client.virtnet /mnt/lustre (opts:) fail_val=2 fail_loc=0x1601 Started LFSCK on the device lustre-MDT0000: scrub namespace Stopped LFSCK on the device lustre-MDT0000. Started LFSCK on the device lustre-MDT0000: scrub namespace fail_loc=0x80001609 Waiting 32s for 'failed' Updated after 3s: want 'failed' got 'failed' fail_loc=0x1600 Started LFSCK on the device lustre-MDT0000: scrub namespace fail_loc=0x160a stop mds1 fail_loc=0x160b start mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 fail_loc=0x1601 Started LFSCK on the device lustre-MDT0000: scrub namespace stop mds1 fail_loc=0x160b start mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 stop mds1 start mds1 without resume LFSCK oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 fail_val=2 fail_loc=0x1602 Started LFSCK on the device lustre-MDT0000: scrub namespace fail_loc=0 fail_val=0 PASS 8 (313s) == sanity-lfsck test 9a: LFSCK speed control (1) ========= 07:19:57 (1788866397) Checking servers environments Checking clients oleg131-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: kmemleak disabled oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: MODOPTS_LIBCFS= oleg131-server: Force libcfs to create 2 CPU partitions oleg131-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg131-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 17 Start of /dev/mapper/mds2_flakey on mds2 failed 17 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg131-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 17 seq.cli-lustre-OST0000-super.width=65536 Start of /dev/mapper/ost1_flakey on ost1 failed 17 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 oleg131-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 17 seq.cli-lustre-OST0001-super.width=65536 Start of /dev/mapper/ost2_flakey on ost2 failed 17 Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre Starting client oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre Started clients oleg131-client.virtnet: 192.168.201.131@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9ca558045800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9ca558045800.idle_timeout=debug disable quota as required - open/close 826 (time 1788866495.24 total 10.03 last 82.37) - open/close 1622 (time 1788866505.25 total 20.03 last 79.59) - open/close 2486 (time 1788866515.25 total 30.04 last 86.33) - open/close 3209 (time 1788866525.26 total 40.04 last 72.28) - open/close 3804 (time 1788866535.26 total 50.04 last 59.47) - open/close 4531 (time 1788866545.27 total 60.05 last 72.65) total: 5000 open/close in 66.83 seconds: 74.82 ops/second Started LFSCK on the device lustre-MDT0000: scrub layout PASS 9a (194s) == sanity-lfsck test 9b: LFSCK speed control (2) ========= 07:23:11 (1788866591) preparing... 0 * 0 files will be created Tue Sep 8 07:23:59 EDT 2026. prepared Tue Sep 8 07:24:03 EDT 2026. Preparing another 50 * 50 files (with error) at Tue Sep 8 07:24:03 EDT 2026. fail_loc=0x1604 total: 50 mkdir in 0.62 seconds: 80.37 ops/second total: 50 create in 0.50 seconds: 100.47 ops/second fail_loc=0x160c Started LFSCK on the device lustre-MDT0000: scrub namespace fail_loc=0 Prepared at Tue Sep 8 07:24:32 EDT 2026. Started LFSCK on the device lustre-MDT0000: scrub namespace PASS 9b (117s) == sanity-lfsck test 10: System is available during LFSCK scanning ========================================================== 07:25:08 (1788866708) preparing... 1 * 1 files will be created Tue Sep 8 07:25:53 EDT 2026. total: 1 mkdir in 0.01 seconds: 154.49 ops/second total: 1 create in 0.00 seconds: 262.41 ops/second total: 1 mkdir in 0.00 seconds: 233.54 ops/second prepared Tue Sep 8 07:25:57 EDT 2026. Preparing more files with error at Tue Sep 8 07:25:57 EDT 2026. fail_loc=0x1603 fail_loc=0x1604 fail_loc=0 Prepared at Tue Sep 8 07:28:18 EDT 2026. 192.168.201.131@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg131-client.virtnet /mnt/lustre (opts:) Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre Started LFSCK on the device lustre-MDT0000: scrub namespace Waiting 90s for 'completed' Waiting 60s for 'completed' Waiting 50s for 'completed' Waiting 40s for 'completed' Waiting 30s for 'completed' Updated after 68s: want 'completed' got 'completed' PASS 10 (362s) == sanity-lfsck test 11a: LFSCK can rebuild lost last_id ========================================================== 07:31:10 (1788867070) total: 64 open/close in 0.89 seconds: 72.18 ops/second stopall remove LAST_ID on ost1: idx=0 removed '/mnt/lustre-ost1/O/0/LAST_ID' oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 fail_val=3 fail_loc=0x160e trigger LFSCK for layout on ost1 to rebuild the LAST_ID(s) Started LFSCK on the device lustre-OST0000: scrub layout fail_val=0 fail_loc=0 Waiting 32s for 'completed' the LAST_ID(s) should have been rebuilt PASS 11a (289s) == sanity-lfsck test 11b: LFSCK can rebuild crashed last_id ========================================================== 07:35:59 (1788867359) Checking servers environments Checking clients oleg131-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: kmemleak disabled oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: MODOPTS_LIBCFS= oleg131-server: Force libcfs to create 2 CPU partitions oleg131-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0000 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0001 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg131-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 17 seq.cli-lustre-OST0000-super.width=65536 Start of /dev/mapper/ost1_flakey on ost1 failed 17 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre Starting client oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre Started clients oleg131-client.virtnet: 192.168.201.131@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9ca547b05800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9ca547b05800.idle_timeout=debug disable quota as required set fail_loc=0x160d to skip the updating LAST_ID on-disk fail_loc=0x160d - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 3329 prealloc_next_id: 3298 count: 32 total: 64 open/close in 0.89 seconds: 72.10 ops/second 192.168.201.131@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg131-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-ost1 (opts:) on oleg131-server fail_loc=0x215 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 the on-disk LAST_ID should be smaller than the expected one trigger LFSCK for layout on ost1 to rebuild the on-disk LAST_ID Started LFSCK on the device lustre-OST0000: scrub layout Stopping /mnt/lustre-ost1 (opts:) on oleg131-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 the on-disk LAST_ID should have been rebuilt fail_loc=0 Stopping clients: oleg131-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg131-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg131-server PASS 11b (187s) == sanity-lfsck test 12a: single command to trigger LFSCK on all devices ========================================================== 07:39:06 (1788867546) Checking servers environments Checking clients oleg131-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: kmemleak disabled oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: MODOPTS_LIBCFS= oleg131-server: Force libcfs to create 2 CPU partitions oleg131-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0000 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0001 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre Starting client oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre Started clients oleg131-client.virtnet: 192.168.201.131@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9ca550818800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9ca550818800.idle_timeout=debug disable quota as required total: 100 open/close in 1.48 seconds: 67.61 ops/second total: 100 open/close in 1.31 seconds: 76.32 ops/second Start namespace LFSCK on all targets by single command (-s 1). Started LFSCK on the device lustre-MDT0000: scrub namespace All the LFSCK targets should be in 'scanning-phase1' status. Stop namespace LFSCK on all targets by single lctl command. Stopped LFSCK on the device lustre-MDT0000. All the LFSCK targets should be in 'stopped' status. Re-start namespace LFSCK on all targets by single command (-s 0). Started LFSCK on the device lustre-MDT0000: scrub namespace All the LFSCK targets should be in 'completed' status. debug=-1 debug_mb=150 debug=-1 debug_mb=150 Start layout LFSCK on all targets by single command (-s 1). Started LFSCK on the device lustre-MDT0000: scrub layout All the LFSCK targets should be in 'scanning-phase1' status. Stop layout LFSCK on all targets by single lctl command. Stopped LFSCK on the device lustre-MDT0000. All the LFSCK targets should be in 'stopped' status. Re-start layout LFSCK on all targets by single command (-s 0). Started LFSCK on the device lustre-MDT0000: scrub layout All the LFSCK targets should be in 'completed' status. debug_mb=21 debug_mb=21 PASS 12a (126s) == sanity-lfsck test 12b: auto detect Lustre device ====== 07:41:12 (1788867672) Start LFSCK without '-M' specified. Started LFSCK on the device lustre-MDT0000: scrub layout namespace Start layout LFSCK on the node with multipe targets, but not specify '-M'/'-A' option. Should get failure. oleg131-server: Detect multiple devices on current node. Please specify the device explicitly via '-M' option or '-A' option for all. pdsh@oleg131-client: oleg131-server: ssh exited with exit code 22 PASS 12b (16s) == sanity-lfsck test 13: LFSCK can repair crashed lmm_oi ========================================================== 07:41:28 (1788867688) ##### The lmm_oi in layout EA should be consistent with the MDT-object FID; otherwise, the LFSCK should re-generate the lmm_oi from the MDT-object FID. ##### Inject failure stub to simulate bad lmm_oi fail_loc=0x160f total: 1 open/close in 0.05 seconds: 21.51 ops/second fail_loc=0 Trigger layout LFSCK to find out the bad lmm_oi and fix them Started LFSCK on the device lustre-MDT0000: scrub layout PASS 13 (12s) == sanity-lfsck test 14a: LFSCK can repair MDT-object with dangling LOV EA reference (1) ========================================================== 07:41:40 (1788867700) ##### The OST-object referenced by the MDT-object should be there; otherwise, the LFSCK should re-create the missing OST-object. without '--delay-create-ostobj' option. ##### Inject failure stub to simulate dangling referenced MDT-object fail_loc=0x1610 - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 3489 prealloc_next_id: 3445 count: 45 total: 61 open/close in 0.84 seconds: 72.45 ops/second touch: setting times of '/mnt/lustre/d14a.sanity-lfsck/guard0': No such file or directory touch: setting times of '/mnt/lustre/d14a.sanity-lfsck/guard1': No such file or directory fail_loc=0 debug=-1 debug_mb=150 debug=-1 debug_mb=150 - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 3553 prealloc_next_id: 3524 count: 30 total: 30 open/close in 1.03 seconds: 29.12 ops/second 'ls' should fail because of dangling referenced MDT-object Trigger layout LFSCK to find out dangling reference Started LFSCK on the device lustre-MDT0000: scrub layout 'stat' should fail because of not repair dangling by default Trigger layout LFSCK to repair dangling reference Started LFSCK on the device lustre-MDT0000: scrub layout 'stat' should success after layout LFSCK repairing debug_mb=21 debug_mb=21 stopall to cleanup object cache setupall oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Using TIMEOUT=20 PASS 14a (200s) == sanity-lfsck test 14b: LFSCK can repair MDT-object with dangling LOV EA reference (2) ========================================================== 07:45:00 (1788867900) ##### The OST-object referenced by the MDT-object should be there; otherwise, the LFSCK should re-create the missing OST-object. with '--delay-create-ostobj' option. ##### Inject failure stub to simulate dangling referenced MDT-object fail_loc=0x1610 - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 3617 prealloc_next_id: 3586 count: 32 total: 63 open/close in 0.93 seconds: 67.91 ops/second touch: setting times of '/mnt/lustre/d14b.sanity-lfsck/guard': No such file or directory fail_loc=0 debug=-1 debug=-1 debug_mb=150 debug_mb=150 - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 3681 prealloc_next_id: 3650 count: 32 total: 32 open/close in 0.94 seconds: 34.09 ops/second 'ls' should fail because of dangling referenced MDT-object Trigger layout LFSCK to find out dangling reference Started LFSCK on the device lustre-MDT0000: scrub layout 'stat' should fail because of not repair dangling by default Trigger layout LFSCK to repair dangling reference Started LFSCK on the device lustre-MDT0000: scrub layout 'stat' should success after layout LFSCK repairing debug_mb=21 debug_mb=21 stopall to cleanup object cache setupall oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Using TIMEOUT=20 PASS 14b (179s) == sanity-lfsck test 15a: LFSCK can repair unmatched MDT-object/OST-object pairs (1) ========================================================== 07:47:59 (1788868079) ##### If the OST-object referenced by the MDT-object back points to some non-exist MDT-object, then the LFSCK should repair the OST-object to back point to the right MDT-object. ##### Inject failure stub to make the OST-object to back point to non-exist MDT-object. fail_loc=0x1611 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675585 s, 15.5 MB/s 257+0 records in 257+0 records out 1052672 bytes (1.1 MB, 1.0 MiB) copied, 0.261692 s, 4.0 MB/s fail_loc=0 Trigger layout LFSCK to find out unmatched pairs and fix them Started LFSCK on the device lustre-MDT0000: scrub layout PASS 15a (14s) == sanity-lfsck test 15b: LFSCK can repair unmatched MDT-object/OST-object pairs (2) ========================================================== 07:48:14 (1788868094) ##### If the OST-object referenced by the MDT-object back points to other MDT-object that doesn't recognize the OST-object, then the LFSCK should repair it to back point to the right MDT-object (the first one). ##### 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703178 s, 14.9 MB/s Inject failure stub to make the OST-object to back point to other MDT-object fail_loc=0x1612 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467494 s, 22.4 MB/s 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.155917 s, 13.5 MB/s fail_loc=0 Trigger layout LFSCK to find out unmatched pairs and fix them Started LFSCK on the device lustre-MDT0000: scrub layout PASS 15b (15s) == sanity-lfsck test 15c: LFSCK can repair unmatched MDT-object/OST-object pairs (3) ========================================================== 07:48:29 (1788868109) SKIP: sanity-lfsck test_15c MDS newer than 2.7.55, LU-6475 SKIP 15c (4s) == sanity-lfsck test 15d: LFSCK don't crash upon dir migration failure ========================================================== 07:48:33 (1788868113) total: 100 open/close in 1.22 seconds: 82.30 ops/second total: 100 mkdir in 0.66 seconds: 151.49 ops/second Migrate /mnt/lustre/d15d.sanity-lfsck to MDT1 fail_loc=0x1709 lfs migrate: cb_migrate_mdt_fini: error completing migration of /mnt/lustre/d15d.sanity-lfsck/s61: Input/output error (5) lfs migrate: /mnt/lustre/d15d.sanity-lfsck/s10 migrate failed: Input/output error (5) lfs migrate: cb_migrate_mdt_fini: error completing migration of /mnt/lustre/d15d.sanity-lfsck: Directory not empty (39) fail_loc=0 fail_loc=0x1709 fail_loc=0 Started LFSCK on the device lustre-MDT0000: scrub namespace lfs migrate: llapi_semantic_traverse: Failed to open '/mnt/lustre/d15d.sanity-lfsck/s61': Input/output error (5) lfs migrate: cb_migrate_mdt_fini: error completing migration of /mnt/lustre/d15d.sanity-lfsck: Directory not empty (39) rm: cannot remove '/mnt/lustre/d15d.sanity-lfsck/s61': Input/output error debug=0 Stopping clients: oleg131-client.virtnet /mnt/lustre (opts:) Stopping client oleg131-client.virtnet /mnt/lustre opts: Stopping clients: oleg131-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg131-server unloading modules via unload_modules_local on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing unload_modules_local oleg131-server: kmemleak disabled modules unloaded. === sanity-lfsck: start setup 07:50:32 (1788868232) === Stopping clients: oleg131-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg131-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 oleg131-server: oleg131-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: kmemleak disabled oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: MODOPTS_LIBCFS= oleg131-server: Force libcfs to create 2 CPU partitions oleg131-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg131-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg131-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg131-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/vdc pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Format mds2: /dev/vdd pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Format ost1: /dev/vde pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Format ost2: /dev/vdf pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Checking servers environments Checking clients oleg131-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: kmemleak disabled oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: MODOPTS_LIBCFS= oleg131-server: Force libcfs to create 2 CPU partitions oleg131-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on /dev/vdc Started lustre-MDT0000 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on /dev/vdd Started lustre-MDT0001 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on /dev/vde Started lustre-OST0000 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on /dev/vdf Started lustre-OST0001 Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre Starting client oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre Started clients oleg131-client.virtnet: 192.168.201.131@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9ca571607000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9ca571607000.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' Updated after 2s: want 'procname_uid' got 'procname_uid' disable quota as required osd-ldiskfs.track_declares_assert=1 === sanity-lfsck: finish setup 07:53:52 (1788868432) === PASS 15d (327s) == sanity-lfsck test 16: LFSCK can repair inconsistent MDT-object/OST-object owner ========================================================== 07:54:00 (1788868440) ##### If the OST-object's owner information does not match the owner information stored in the MDT-object, then the LFSCK trust the MDT-object and update the OST-object's owner information. ##### 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258599 s, 40.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [createmany] [-o] [/mnt/lustre/d16.sanity-lfsck/d1/o] [100] total: 100 open/close in 1.56 seconds: 63.92 ops/second Inject failure stub to skip OST-object owner changing fail_loc=0x1613 fail_loc=0 Trigger layout LFSCK to find out inconsistent OST-object owner and fix them Started LFSCK on the device lustre-MDT0000: scrub layout PASS 16 (15s) == sanity-lfsck test 17: LFSCK can repair multiple references ========================================================== 07:54:16 (1788868456) ##### If more than one MDT-objects reference the same OST-object, and the OST-object only recognizes one MDT-object, then the LFSCK should create new OST-objects for such non-recognized MDT-objects. ##### Inject failure stub to make two MDT-objects to refernce the OST-object fail_val=0 fail_loc=0x1614 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598292 s, 17.5 MB/s total: 1 open/close in 0.04 seconds: 28.33 ops/second fail_loc=0 fail_val=0 /mnt/lustre/d17.sanity-lfsck/f0 and /mnt/lustre/d17.sanity-lfsck/guard use the same OST-objects /mnt/lustre/d17.sanity-lfsck/f1 and /mnt/lustre/d17.sanity-lfsck/guard use the same OST-objects Trigger layout LFSCK to find out multiple refenced MDT-objects and fix them Started LFSCK on the device lustre-MDT0000: scrub layout /mnt/lustre/d17.sanity-lfsck/f0 and /mnt/lustre/d17.sanity-lfsck/guard should use diff OST-objects 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.098459 s, 21.3 MB/s /mnt/lustre/d17.sanity-lfsck/f1 and /mnt/lustre/d17.sanity-lfsck/guard should use diff OST-objects 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0637865 s, 32.9 MB/s PASS 17 (14s) == sanity-lfsck test 18a: Find out orphan OST-object and repair it (1) ========================================================== 07:54:30 (1788868470) ##### The target MDT-object is there, but related stripe information is lost or partly lost. The LFSCK should regenerate the missing layout EA entries. ##### 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0840489 s, 25.0 MB/s [0x200000402:0x73:0x0] /mnt/lustre/d18a.sanity-lfsck/a1/f1 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 108 0x6c 0x280000401 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.208848 s, 10.0 MB/s [0x240000402:0x2:0x0] /mnt/lustre/d18a.sanity-lfsck/a2/f2 lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2 0x2 0x2c0000400 0 2 0x2 0x280000400 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.14735 s, 14.2 MB/s [0x200000402:0x75:0x0] /mnt/lustre/d18a.sanity-lfsck/f3 lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6d:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2:0x0] } Inject failure, to make the MDT-object lost its layout EA fail_loc=0x1615 fail_loc=0x1615 fail_loc=0 fail_loc=0 The file size should be incorrect since layout EA is lost Trigger layout LFSCK on all devices to find out orphan OST-object Started LFSCK on the device lustre-MDT0000: scrub layout [0x200000402:0x73:0x0] /mnt/lustre/d18a.sanity-lfsck/a1/f1 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 1 lmm_stripe_offset: 0 obdidx objid objid group 0 108 0x6c 0x280000401 [0x240000402:0x2:0x0] /mnt/lustre/d18a.sanity-lfsck/a2/f2 lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 2 lmm_stripe_offset: 1 obdidx objid objid group 1 2 0x2 0x2c0000400 0 2 0x2 0x280000400 [0x200000402:0x75:0x0] /mnt/lustre/d18a.sanity-lfsck/f3 lcm_layout_gen: 1 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 1 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6d:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 1 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2:0x0] } The file size should be correct after layout LFSCK scanning PASS 18a (25s) SKIP: sanity-lfsck test_18b skipping excluded test 18b == sanity-lfsck test 18c: Find out orphan OST-object and repair it (3) ========================================================== 07:54:57 (1788868497) ##### The target MDT-object is lost, and the OST-object FID is missing. The LFSCK should re-create the MDT-object with new FID under the directory .lustre/lost+found/MDTxxxx. ##### Inject failure, to simulate the case of missing parent FID fail_loc=0x1617 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109276 s, 19.2 MB/s /mnt/lustre/d18c.sanity-lfsck/a1/f1 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 110 0x6e 0x280000401 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.102881 s, 20.4 MB/s /mnt/lustre/d18c.sanity-lfsck/a2/f2 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3 0x3 0x280000400 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.195309 s, 10.7 MB/s /mnt/lustre/d18c.sanity-lfsck/f3 lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6f:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x3:0x0] } fail_loc=0 Inject failure, to simulate the case of missing the MDT-object fail_loc=0x1616 fail_loc=0x1616 fail_loc=0 fail_loc=0 Trigger layout LFSCK on all devices to find out orphan OST-object Started LFSCK on the device lustre-MDT0000: scrub layout total 12 144115188109410307 dr-x------ 4 root root 4096 Dec 31 1969 . 144115205306056705 drwx------ 2 root root 4096 Sep 8 07:54 MDT0000 162129603815538689 drwx------ 2 root root 4096 Sep 8 07:54 MDT0001 There should NOT be some stub under .lustre/lost+found/MDT0001/ There should be some stub under .lustre/lost+found/MDT0000/ PASS 18c (28s) == sanity-lfsck test 18d: Find out orphan OST-object and repair it (4) ========================================================== 07:55:26 (1788868526) ##### The target MDT-object layout EA is corrupted, but the right OST-object is still alive as orphan. The layout LFSCK will not create new OST-object to occupy such slot. ##### [0x200000402:0x81:0x0] /mnt/lustre/d18d.sanity-lfsck/a1/f1 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 112 0x70 0x280000401 [0x200000402:0x82:0x0] /mnt/lustre/d18d.sanity-lfsck/a1/f2 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 113 0x71 0x280000401 [0x200000402:0x83:0x0] /mnt/lustre/d18d.sanity-lfsck/a1/f3 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 114 0x72 0x280000401 [0x200000402:0x84:0x0] /mnt/lustre/d18d.sanity-lfsck/a1/f4 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x73:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 Inject failure to make /mnt/lustre/d18d.sanity-lfsck/a1/f1 and /mnt/lustre/d18d.sanity-lfsck/a1/f2 to reference the same OST-object (which is f1's OST-object). Then drop /mnt/lustre/d18d.sanity-lfsck/a1/f1 and its OST-object, so f2 becomes dangling reference case, but f2's old OST-object is there. The failure also makes /mnt/lustre/d18d.sanity-lfsck/a1/f3 and /mnt/lustre/d18d.sanity-lfsck/a1/f4 to reference the same OST-object (which is f3's OST-object). Then drop /mnt/lustre/d18d.sanity-lfsck/a1/f3 and its OST-object, so f4 becomes dangling reference case, but f4's old OST-object is there. fail_loc=0x1618 fail_loc=0 stopall to cleanup object cache setupall oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Using TIMEOUT=20 Trigger layout LFSCK on all devices to find out orphan OST-object Started LFSCK on the device lustre-MDT0000: scrub layout The file size should be correct after layout LFSCK scanning The LFSCK should find back the original data. foo [0x200000402:0x82:0x0] /mnt/lustre/d18d.sanity-lfsck/a1/f2 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 1 lmm_stripe_offset: 0 obdidx objid objid group 0 113 0x71 0x280000401 foo [0x200000402:0x84:0x0] /mnt/lustre/d18d.sanity-lfsck/a1/f4 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 1 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x73:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 18d (185s) == sanity-lfsck test 18e: Find out orphan OST-object and repair it (5) ========================================================== 07:58:31 (1788868711) ##### The target MDT-object layout EA slot is occpuied by some new created OST-object when repair dangling reference case. Such conflict OST-object has been modified by others. To keep the new data, the LFSCK will create a new file to refernece this old orphan OST-object. ##### [0x200000bd1:0x4:0x0] /mnt/lustre/d18e.sanity-lfsck/a1/f1 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 162 0xa2 0x280000401 [0x200000bd1:0x5:0x0] /mnt/lustre/d18e.sanity-lfsck/a1/f2 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 163 0xa3 0x280000401 [0x200000bd1:0x6:0x0] /mnt/lustre/d18e.sanity-lfsck/a1/f3 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 164 0xa4 0x280000401 [0x200000bd1:0x7:0x0] /mnt/lustre/d18e.sanity-lfsck/a1/f4 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa5:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 Inject failure to make /mnt/lustre/d18e.sanity-lfsck/a1/f1 and /mnt/lustre/d18e.sanity-lfsck/a1/f2 to reference the same OST-object (which is f1's OST-object). Then drop /mnt/lustre/d18e.sanity-lfsck/a1/f1 and its OST-object, so f2 becomes dangling reference case, but f2's old OST-object is there. Also the failure makes /mnt/lustre/d18e.sanity-lfsck/a1/f3 and /mnt/lustre/d18e.sanity-lfsck/a1/f4 to reference the same OST-object (which is f3's OST-object). Then drop /mnt/lustre/d18e.sanity-lfsck/a1/f3 and its OST-object, so f4 becomes dangling reference case, but f4's old OST-object is there. fail_loc=0x1618 fail_loc=0 stopall to cleanup object cache