== sanity-lfsck test 18f: Skip the failed OST(s) when handle orphan OST-objects ========================================================== 20:51:36 (1767318696) ##### The target MDT-object is lost. The LFSCK should re-create the MDT-object under .lustre/lost+found/MDTxxxx. If some OST fail to verify some OST-object(s) during the first stage-scanning, the LFSCK should skip orphan OST-objects for such OST. Others should not be affected. ##### 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0812217 s, 25.8 MB/s 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.064405 s, 32.6 MB/s 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0635803 s, 33.0 MB/s /mnt/lustre/d18f.sanity-lfsck/a1/f1 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3299 0xce3 0x280000400 /mnt/lustre/d18f.sanity-lfsck/a2/f2 lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3300 0xce4 0x280000400 1 2850 0xb22 0x240000400 Inject failure, to simulate the case of missing the MDT-object fail_loc=0x1616 fail_loc=0 Inject failure, to simulate the OST0 fail to handle MDT0 LFSCK request during the first-stage scanning. fail_loc=0x161c fail_val=0 Trigger layout LFSCK on all devices to find out orphan OST-object Started LFSCK on the device lustre-MDT0000: scrub layout oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 120s for 'partial' oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 100s for 'partial' oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 60s for 'partial' oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 30s for 'partial' oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg423-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 0s for 'partial' Update not seen after 120s: want 'partial' got '' sanity-lfsck test_18f: @@@@@@ FAIL: (2) MDS1 is not the expected 'partial' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7487:error() = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:2881:test_18f() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7863:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7741:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:2948:main() Dumping lctl log to /tmp/testlogs//sanity-lfsck.test_18f.*.1767318828.log rsync: chown "/tmp/testlogs/.sanity-lfsck.test_18f.debug_log.oleg423-server.1767318828.log.5SMQyp" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lfsck.test_18f.dmesg.oleg423-server.1767318828.log.h0IUQB" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]