== sanity-lfsck test 20a: Handle the orphan with dummy LOV EA slot properly ========================================================== 21:04:36 (1767319476) ##### The target MDT-object and some of its OST-object are lost. The LFSCK should find out the left OST-objects and re-create the MDT-object under the direcotry .lustre/lost+found/MDTxxxx/ with the partial OST-objects (LOV EA hole). New client can access the file with LOV EA hole via normal system tools or commands without crash the system. For old client, even though it cannot access the file with LOV EA hole, it should not cause the system crash. ##### 257+0 records in 257+0 records out 1052672 bytes (1.1 MB, 1.0 MiB) copied, 0.1764 s, 6.0 MB/s 257+0 records in 257+0 records out 1052672 bytes (1.1 MB, 1.0 MiB) copied, 0.247759 s, 4.2 MB/s 257+0 records in 257+0 records out 1052672 bytes (1.1 MB, 1.0 MiB) copied, 0.292515 s, 3.6 MB/s [0x200004a51:0x21:0x0] /mnt/lustre/d20a.sanity-lfsck/a1/f0 lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3308 0xcec 0x240000400 1 2854 0xb26 0x280000400 [0x200004a51:0x22:0x0] /mnt/lustre/d20a.sanity-lfsck/a1/f1 lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3309 0xced 0x240000400 1 2855 0xb27 0x280000400 [0x200004a51:0x23:0x0] /mnt/lustre/d20a.sanity-lfsck/a1/f2 lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3310 0xcee 0x240000400 1 2856 0xb28 0x280000400 Inject failure... To simulate f0 lost MDT-object fail_loc=0x1616 To simulate f1 lost MDT-object and OST-object0 fail_loc=0x161a To simulate f2 lost MDT-object and OST-object1 fail_val=1 192.168.203.141@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg341-client.virtnet /mnt/lustre (opts:) fail_loc=0 fail_val=0 Trigger layout LFSCK on all devices to find out orphan OST-object Started LFSCK on the device lustre-MDT0000: scrub layout oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 32s for 'completed' oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 22s for 'completed' oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 2s for 'completed' oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Update not seen after 32s: want 'completed' got '' sanity-lfsck test_20a: @@@@@@ FAIL: (2) MDS1 is not the expected 'completed' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7487:error() = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:3265:test_20a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7863:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7741:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:3512:main() Dumping lctl log to /tmp/testlogs//sanity-lfsck.test_20a.*.1767319530.log rsync: chown "/tmp/testlogs/.sanity-lfsck.test_20a.debug_log.oleg341-server.1767319530.log.lw4HP3" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lfsck.test_20a.dmesg.oleg341-server.1767319530.log.e3d974" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]