== sanity-lfsck test 18a: Find out orphan OST-object and repair it (1) ========================================================== 20:36:48 (1767317808) ##### The target MDT-object is there, but related stripe information is lost or partly lost. The LFSCK should regenerate the missing layout EA entries. ##### 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0487753 s, 43.0 MB/s [0x200003ab1:0x80:0x0] /mnt/lustre/d18a.sanity-lfsck/a1/f1 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3217 0xc91 0x280000400 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100447 s, 20.9 MB/s [0x200003ab1:0x81:0x0] /mnt/lustre/d18a.sanity-lfsck/f3 lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xc92:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x240000400:0xae6:0x0] } Inject failure, to make the MDT-object lost its layout EA fail_loc=0x1615 fail_loc=0 The file size should be incorrect since layout EA is lost Trigger layout LFSCK on all devices to find out orphan OST-object Started LFSCK on the device lustre-MDT0000: scrub layout oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 120s for 'completed' oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 110s for 'completed' oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 100s for 'completed' oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 80s for 'completed' oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 70s for 'completed' oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 60s for 'completed' oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 40s for 'completed' oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 30s for 'completed' oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 20s for 'completed' oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 10s for 'completed' oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 0s for 'completed' Update not seen after 120s: want 'completed' got '' sanity-lfsck test_18a: @@@@@@ FAIL: (4) MDS1 is not the expected 'completed' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7487:error() = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:2207:test_18a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7863:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7741:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:2258:main() Dumping lctl log to /tmp/testlogs//sanity-lfsck.test_18a.*.1767317938.log rsync: chown "/tmp/testlogs/.sanity-lfsck.test_18a.debug_log.oleg423-server.1767317938.log.kjguj9" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lfsck.test_18a.dmesg.oleg423-server.1767317938.log.npySZW" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]