== sanity-lfsck test 18e: Find out orphan OST-object and repair it (5) ========================================================== 20:50:55 (1767318655) ##### The target MDT-object layout EA slot is occpuied by some new created OST-object when repair dangling reference case. Such conflict OST-object has been modified by others. To keep the new data, the LFSCK will create a new file to refernece this old orphan OST-object. ##### [0x200004281:0x4:0x0] /mnt/lustre/d18e.sanity-lfsck/a1/f1 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3266 0xcc2 0x240000400 [0x200004281:0x5:0x0] /mnt/lustre/d18e.sanity-lfsck/a1/f2 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3267 0xcc3 0x240000400 [0x200004281:0x6:0x0] /mnt/lustre/d18e.sanity-lfsck/a1/f3 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3268 0xcc4 0x240000400 [0x200004281:0x7:0x0] /mnt/lustre/d18e.sanity-lfsck/a1/f4 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcc5:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 Inject failure to make /mnt/lustre/d18e.sanity-lfsck/a1/f1 and /mnt/lustre/d18e.sanity-lfsck/a1/f2 to reference the same OST-object (which is f1's OST-object). Then drop /mnt/lustre/d18e.sanity-lfsck/a1/f1 and its OST-object, so f2 becomes dangling reference case, but f2's old OST-object is there. Also the failure makes /mnt/lustre/d18e.sanity-lfsck/a1/f3 and /mnt/lustre/d18e.sanity-lfsck/a1/f4 to reference the same OST-object (which is f3's OST-object). Then drop /mnt/lustre/d18e.sanity-lfsck/a1/f3 and its OST-object, so f4 becomes dangling reference case, but f4's old OST-object is there. fail_loc=0x1618 fail_loc=0 stopall to cleanup object cache setupall /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg341-server: oleg341-server.virtnet: executing load_modules_local oleg341-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg341-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg341-server: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 oleg341-server: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg341-server: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 oleg341-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 oleg341-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Using TIMEOUT=20 fail_val=10 fail_loc=0x1602 debug=-1 debug_mb=150 debug=-1 debug_mb=150 Trigger layout LFSCK on all devices to find out orphan OST-object Started LFSCK on the device lustre-MDT0000: scrub layout oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 120s for 'scanning-phase2' oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 110s for 'scanning-phase2' oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 80s for 'scanning-phase2' oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 50s for 'scanning-phase2' oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 30s for 'scanning-phase2' oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 0s for 'scanning-phase2' Update not seen after 120s: want 'scanning-phase2' got '' sanity-lfsck test_18e: @@@@@@ FAIL: (3) MDS1 is not the expected 'scanning-phase2' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7487:error() = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:2730:test_18e() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7863:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7741:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:2805:main() Dumping lctl log to /tmp/testlogs//sanity-lfsck.test_18e.*.1767318995.log rsync: chown "/tmp/testlogs/.sanity-lfsck.test_18e.debug_log.oleg341-server.1767318995.log.DK9mQf" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lfsck.test_18e.dmesg.oleg341-server.1767318995.log.nda51g" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]