== sanity-lfsck test 18h: LFSCK can repair crashed PFL extent range ========================================================== 21:01:38 (1767319298) ##### The PFL extent crashed. During the first cycle LFSCK scanning, the layout LFSCK will keep the bad PFL file(s) there without scanning its OST-object(s). Then in the second stage scanning, the OST will return related OST-object(s) to the MDT as orphan. And then the LFSCK on the MDT can rebuild the PFL extent with the 'orphan(s)' stripe information. ##### 0+1 records in 0+1 records out 316142 bytes (316 kB, 309 KiB) copied, 0.0145222 s, 21.8 MB/s Inject failure stub to simulate bad PFL extent range fail_loc=0x162f fail_loc=0 dd: error writing '/mnt/lustre/d18h.sanity-lfsck/f0': No data available 1+0 records in 0+0 records out 0 bytes copied, 0.0227045 s, 0.0 kB/s Trigger layout LFSCK to find out the bad lmm_oi and fix them Started LFSCK on the device lustre-MDT0000: scrub layout oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 120s for 'completed' oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 110s for 'completed' oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 100s for 'completed' oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 90s for 'completed' oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 80s for 'completed' oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 70s for 'completed' oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 60s for 'completed' oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 20s for 'completed' oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 10s for 'completed' oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 0s for 'completed' Update not seen after 120s: want 'completed' got '' sanity-lfsck test_18h: @@@@@@ FAIL: (4.1) MDS1 is not the expected 'completed' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7487:error() = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:3059:test_18h() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7863:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7741:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:3084:main() Dumping lctl log to /tmp/testlogs//sanity-lfsck.test_18h.*.1767319432.log rsync: chown "/tmp/testlogs/.sanity-lfsck.test_18h.debug_log.oleg341-server.1767319432.log.nPsCof" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lfsck.test_18h.dmesg.oleg341-server.1767319432.log.fT8s3L" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]