== sanity-lfsck test 20b: Handle the orphan with dummy LOV EA slot properly - PFL case ========================================================== 21:05:43 (1767319543) ##### The target MDT-object and some of its OST-object are lost. The LFSCK should find out the left OST-objects and re-create the MDT-object under the direcotry .lustre/lost+found/MDTxxxx/ with the partial OST-objects (LOV EA hole). New client can access the file with LOV EA hole via normal system tools or commands without crash the system - PFL case. ##### Checking servers environments Checking clients oleg341-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing load_modules_local oleg341-server: Loading modules from /home/green/git/lustre-release/lustre oleg341-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg341-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg341-server: detected 4 online CPUs by sysfs oleg341-server: MODOPTS_LIBCFS= oleg341-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg341-server: mount.lustre: according to /etc/mtab lustre-mdt1/mdt1 is already mounted on /mnt/lustre-mds1 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 17 Start of lustre-mdt1/mdt1 on mds1 failed 17 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 oleg341-server: mount.lustre: according to /etc/mtab lustre-ost1/ost1 is already mounted on /mnt/lustre-ost1 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 17 oleg341-server: mount.lustre: according to /etc/mtab lustre-ost2/ost2 is already mounted on /mnt/lustre-ost2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 17 seq.cli-lustre-OST0000-super.width=65536 Start of lustre-ost1/ost1 on ost1 failed 17 seq.cli-lustre-OST0001-super.width=65536 Start of lustre-ost2/ost2 on ost2 failed 17 Starting client: oleg341-client.virtnet: -o user_xattr,flock 192.168.203.141@tcp:/lustre /mnt/lustre Starting client oleg341-client.virtnet: -o user_xattr,flock 192.168.203.141@tcp:/lustre /mnt/lustre Started clients oleg341-client.virtnet: 192.168.203.141@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff90cb0770a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff90cb0770a800.idle_timeout=debug disable quota as required 769+0 records in 769+0 records out 3149824 bytes (3.1 MB, 3.0 MiB) copied, 0.915419 s, 3.4 MB/s 769+0 records in 769+0 records out 3149824 bytes (3.1 MB, 3.0 MiB) copied, 1.0129 s, 3.1 MB/s 769+0 records in 769+0 records out 3149824 bytes (3.1 MB, 3.0 MiB) copied, 0.709269 s, 4.4 MB/s [0x200004a52:0x2:0x0] /mnt/lustre/d20b.sanity-lfsck/f0 lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcef:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0xb29:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb2c:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0xcf2:0x0] } [0x200004a52:0x3:0x0] /mnt/lustre/d20b.sanity-lfsck/f1 lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb2a:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0xcf0:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcf3:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0xb2d:0x0] } [0x200004a52:0x4:0x0] /mnt/lustre/d20b.sanity-lfsck/f2 lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcf1:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0xb2b:0x0] } lcme_id: 2 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb2e:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0xcf4:0x0] } Inject failure... To simulate f0 lost MDT-object fail_loc=0x1616 To simulate the case of f1 lost MDT-object and the first OST-object in each PFL component fail_loc=0x161a To simulate the case of f2 lost MDT-object and the second OST-object in each PFL component fail_val=1 fail_loc=0 fail_val=0 Trigger layout LFSCK on all devices to find out orphan OST-object Started LFSCK on the device lustre-MDT0000: scrub layout oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 32s for 'completed' oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Waiting 12s for 'completed' oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory oleg341-server: error: get_param: param_path 'mdd//lfsck_layout': No such file or directory Update not seen after 32s: want 'completed' got '' sanity-lfsck test_20b: @@@@@@ FAIL: (4) MDS1 is not the expected 'completed' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7487:error() = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:3590:test_20b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7863:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7741:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:3824:main() Dumping lctl log to /tmp/testlogs//sanity-lfsck.test_20b.*.1767319717.log rsync: chown "/tmp/testlogs/.sanity-lfsck.test_20b.debug_log.oleg341-server.1767319717.log.B5Pote" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lfsck.test_20b.dmesg.oleg341-server.1767319717.log.OIwGIx" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]