== sanity-lfsck test 18d: Find out orphan OST-object and repair it (4) ========================================================== 20:41:35 (1767318095) ##### The target MDT-object layout EA is corrupted, but the right OST-object is still alive as orphan. The layout LFSCK will not create new OST-object to occupy such slot. ##### [0x200003ab1:0x8c:0x0] /mnt/lustre/d18d.sanity-lfsck/a1/f1 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3221 0xc95 0x280000400 [0x200003ab1:0x8d:0x0] /mnt/lustre/d18d.sanity-lfsck/a1/f2 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3222 0xc96 0x280000400 [0x200003ab1:0x8e:0x0] /mnt/lustre/d18d.sanity-lfsck/a1/f3 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3223 0xc97 0x280000400 [0x200003ab1:0x8f:0x0] /mnt/lustre/d18d.sanity-lfsck/a1/f4 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xc98:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 Inject failure to make /mnt/lustre/d18d.sanity-lfsck/a1/f1 and /mnt/lustre/d18d.sanity-lfsck/a1/f2 to reference the same OST-object (which is f1's OST-object). Then drop /mnt/lustre/d18d.sanity-lfsck/a1/f1 and its OST-object, so f2 becomes dangling reference case, but f2's old OST-object is there. The failure also makes /mnt/lustre/d18d.sanity-lfsck/a1/f3 and /mnt/lustre/d18d.sanity-lfsck/a1/f4 to reference the same OST-object (which is f3's OST-object). Then drop /mnt/lustre/d18d.sanity-lfsck/a1/f3 and its OST-object, so f4 becomes dangling reference case, but f4's old OST-object is there. fail_loc=0x1618 fail_loc=0 stopall to cleanup object cache setupall /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg423-server: oleg423-server.virtnet: executing load_modules_local oleg423-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg423-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 oleg423-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 oleg423-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Using TIMEOUT=20 Trigger layout LFSCK on all devices to find out orphan OST-object Started LFSCK on the device lustre-MDT0000: scrub layout oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 120s for 'completed' oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 90s for 'completed' oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 70s for 'completed' oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 60s for 'completed' oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 40s for 'completed' oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 20s for 'completed' oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 0s for 'completed' Update not seen after 120s: want 'completed' got '' sanity-lfsck test_18d: @@@@@@ FAIL: (3) MDS1 is not the expected 'completed' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7487:error() = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:2612:test_18d() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7863:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7741:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:2652:main() Dumping lctl log to /tmp/testlogs//sanity-lfsck.test_18d.*.1767318375.log rsync: chown "/tmp/testlogs/.sanity-lfsck.test_18d.debug_log.oleg423-server.1767318375.log.jqqbss" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lfsck.test_18d.dmesg.oleg423-server.1767318375.log.HXjVb9" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]