== sanity-lfsck test 8: LFSCK state machine ============== 12:34:43 (1736357683) formatall oleg111-server: oleg111-server.virtnet: executing set_hostid libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory setupall oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 oleg111-server: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 oleg111-server: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Using TIMEOUT=20 preparing... 20 * 20 files will be created Wed Jan 8 12:35:21 EST 2025. total: 20 mkdir in 0.03 seconds: 696.26 ops/second total: 20 create in 0.03 seconds: 677.02 ops/second total: 20 mkdir in 0.03 seconds: 742.70 ops/second prepared Wed Jan 8 12:35:23 EST 2025. fail_loc=0x1603 fail_loc=0x1604 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) fail_val=2 fail_loc=0x1601 Started LFSCK on the device lustre-MDT0000: scrub namespace Stopped LFSCK on the device lustre-MDT0000. Started LFSCK on the device lustre-MDT0000: scrub namespace fail_loc=0x80001609 Waiting 32s for 'failed' Updated after 3s: want 'failed' got 'failed' fail_loc=0x1600 Started LFSCK on the device lustre-MDT0000: scrub namespace fail_loc=0x160a stop mds1 fail_loc=0x160b start mds1 oleg111-server: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 fail_loc=0x1601 Started LFSCK on the device lustre-MDT0000: scrub namespace stop mds1 fail_loc=0x160b start mds1 oleg111-server: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 stop mds1 start mds1 without resume LFSCK oleg111-server: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 fail_val=2 fail_loc=0x1602 Started LFSCK on the device lustre-MDT0000: scrub namespace name: lfsck_namespace magic: 0xa06249ff version: 2 status: scanning-phase2 flags: scanned-once param: last_completed_time: N/A time_since_last_completed: N/A latest_start_time: 1736357753 time_since_latest_start: 3 seconds last_checkpoint_time: 1736357753 time_since_last_checkpoint: 3 seconds latest_start_position: 539, [0x200000007:0x1:0x0], 0x0 last_checkpoint_position: 35184372088832, N/A, N/A first_failure_position: N/A, N/A, N/A checked_phase1: 445 checked_phase2: 1 updated_phase1: 0 updated_phase2: 0 failed_phase1: 0 failed_phase2: 0 directories: 43 dirent_repaired: 0 linkea_repaired: 0 nlinks_repaired: 0 multiple_linked_checked: 0 multiple_linked_repaired: 0 unknown_inconsistency: 0 unmatched_pairs_repaired: 0 dangling_repaired: 0 multiple_referenced_repaired: 0 bad_file_type_repaired: 0 lost_dirent_repaired: 0 local_lost_found_scanned: 0 local_lost_found_moved: 0 local_lost_found_skipped: 0 local_lost_found_failed: 0 striped_dirs_scanned: 0 striped_dirs_repaired: 0 striped_dirs_failed: 0 striped_dirs_disabled: 0 striped_dirs_skipped: 0 striped_shards_scanned: 0 striped_shards_repaired: 0 striped_shards_failed: 0 striped_shards_skipped: 0 name_hash_repaired: 0 linkea_overflow_cleared: 0 agent_entries_repaired: 0 success_count: 0 run_time_phase1: 6 seconds run_time_phase2: 3 seconds average_speed_phase1: 74 items/sec average_speed_phase2: 0 objs/sec average_speed_total: 49 items/sec real_time_speed_phase1: N/A real_time_speed_phase2: 0 objs/sec current_position: [0x200000401:0x221:0x0] sanity-lfsck test_8: @@@@@@ FAIL: (23) Expect 'scanned-once,inconsistent',but got 'scanned-once' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6421:error() = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:969:namespace_error() = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:1153:test_8() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6756:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6803:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6644:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:1167:main() Dumping lctl log to /tmp/testlogs//sanity-lfsck.test_8.*.1736357756.log rsync: chown "/tmp/testlogs/.sanity-lfsck.test_8.debug_log.oleg111-server.1736357756.log.7TI6P1" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lfsck.test_8.dmesg.oleg111-server.1736357756.log.oUHgAy" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]