== sanity-lfsck test 20b: Handle the orphan with dummy LOV EA slot properly - PFL case ========================================================== 20:29:34 (1770686974) ##### The target MDT-object and some of its OST-object are lost. The LFSCK should find out the left OST-objects and re-create the MDT-object under the direcotry .lustre/lost+found/MDTxxxx/ with the partial OST-objects (LOV EA hole). New client can access the file with LOV EA hole via normal system tools or commands without crash the system - PFL case. ##### 769+0 records in 769+0 records out 3149824 bytes (3.1 MB, 3.0 MiB) copied, 0.359463 s, 8.8 MB/s 769+0 records in 769+0 records out 3149824 bytes (3.1 MB, 3.0 MiB) copied, 0.339037 s, 9.3 MB/s 769+0 records in 769+0 records out 3149824 bytes (3.1 MB, 3.0 MiB) copied, 0.340032 s, 9.3 MB/s [0x200004a52:0x3:0x0] /mnt/lustre/d20b.sanity-lfsck/f0 lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcef:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0xb29:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb2c:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0xcf2:0x0] } [0x200004a52:0x4:0x0] /mnt/lustre/d20b.sanity-lfsck/f1 lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb2a:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0xcf0:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcf3:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0xb2d:0x0] } [0x200004a52:0x5:0x0] /mnt/lustre/d20b.sanity-lfsck/f2 lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcf1:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0xb2b:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb2e:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0xcf4:0x0] } Inject failure... To simulate f0 lost MDT-object fail_loc=0x1616 To simulate the case of f1 lost MDT-object and the first OST-object in each PFL component fail_loc=0x161a To simulate the case of f2 lost MDT-object and the second OST-object in each PFL component fail_val=1 fail_loc=0 fail_val=0 Trigger layout LFSCK on all devices to find out orphan OST-object Started LFSCK on the device lustre-MDT0000: scrub layout Check /mnt/lustre/.lustre/lost+found/MDT0000/[0x200004a52:0x3:0x0]-R-0, which is the old f0 /mnt/lustre/.lustre/lost+found/MDT0000/[0x200004a52:0x3:0x0]-R-0 composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 288 lcm_flags: 0 lcm_layout_gen: 1 lcm_mirror_count: 1 lcm_entry_count: 2 components: - lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lcme_offset: 128 lcme_size: 80 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200004a52 lmm_object_id: 0x3 lmm_fid: [0x200004a52:0x3:0x0] lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 2 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcef:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0xb29:0x0] } - lcme_id: 2 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: EOF lcme_offset: 208 lcme_size: 80 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200004a52 lmm_object_id: 0x3 lmm_fid: [0x200004a52:0x3:0x0] lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 2 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb2c:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0xcf2:0x0] } Check /mnt/lustre/.lustre/lost+found/MDT0000/[0x200004a52:0x4:0x0]-R-0, it contains f1's second OST-object in each COMP /mnt/lustre/.lustre/lost+found/MDT0000/[0x200004a52:0x4:0x0]-R-0 composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 288 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 2 components: - lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lcme_offset: 128 lcme_size: 80 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200004a52 lmm_object_id: 0x4 lmm_fid: [0x200004a52:0x4:0x0] lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0,hole lmm_layout_gen: 1 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x0:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0xcf0:0x0] } - lcme_id: 2 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: EOF lcme_offset: 208 lcme_size: 80 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200004a52 lmm_object_id: 0x4 lmm_fid: [0x200004a52:0x4:0x0] lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0,hole lmm_layout_gen: 1 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x0:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0xb2d:0x0] } sanity-lfsck test_20b: @@@@@@ FAIL: (8.2.1) expect pattern flag hole, but got raid0,hole Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7319:error() = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:3675:test_20b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7695:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7758:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7573:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:3824:main() Dumping lctl log to /tmp/testlogs//sanity-lfsck.test_20b.*.1770686988.log rsync: chown "/tmp/testlogs/.sanity-lfsck.test_20b.debug_log.oleg416-server.1770686988.log.D1PCjN" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lfsck.test_20b.dmesg.oleg416-server.1770686988.log.KRSwxo" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]