-----============= acceptance-small: sanity-flr ============----- Sun Nov 16 21:00:42 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg308-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 21:00:58 (1763344858) === oleg308-client.virtnet: executing check_config_client /mnt/lustre oleg308-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg308-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff958d8812a000.idle_timeout=debug osc.lustre-OST0001-osc-ffff958d8812a000.idle_timeout=debug disable quota as required oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 21:01:27 (1763344887) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8225] == sanity-flr test 0a: lfs mirror create with -N option == 21:01:29 (1763344889) PASS 0a (12s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 21:01:41 (1763344901) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 21:01:43 (1763344903) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 21:01:46 (1763344906) PASS 0d (17s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 21:02:03 (1763344923) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 21:02:05 (1763344925) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 21:02:08 (1763344928) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff958d8812a000.stats=clear osc.lustre-OST0001-osc-ffff958d8812a000.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 21:02:14 (1763344934) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (8s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 21:02:22 (1763344942) PASS 0j (5s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 21:02:27 (1763344947) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 21:02:29 (1763344949) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (7s) == sanity-flr test 2: create components from existing files ========================================================== 21:02:36 (1763344956) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 21:02:42 (1763344962) PASS 3 (5s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 21:02:47 (1763344967) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 21:02:53 (1763344973) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (5s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 21:03:01 (1763344981) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 21:03:07 (1763344987) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.187475 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801812 s, 13.1 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 21:03:15 (1763344995) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.214431 s, 44.0 MB/s PASS 22 (7s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 21:03:21 (1763345001) Stopping /mnt/lustre-ost1 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff958d8812a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff958d8812a000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg308-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff958d8812a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff958d8812a000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg308-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (59s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 21:04:20 (1763345060) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.273804 s, 30.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff958d8812a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff958d8812a000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg308-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (29s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 21:04:49 (1763345089) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff958d8812a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff958d8812a000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg308-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff958d8812a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff958d8812a000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg308-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (60s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 21:05:49 (1763345149) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.23353 s, 23.5 MB/s -rw-r--r-- 1 root root 52428800 Nov 16 21:05 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.729s user 0m0.001s sys 0m0.152s Stopping /mnt/lustre-ost1 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff958d8812a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff958d8812a000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.663s user 0m0.000s sys 0m0.155s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg308-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff958d8812a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff958d8812a000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.805s user 0m0.000s sys 0m0.143s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg308-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg308-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff958d8812a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff958d8812a000.ost_server_uuid in FULL state after 0 sec PASS 33b (67s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 21:06:56 (1763345216) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 21:06:59 (1763345219) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 21:07:02 (1763345222) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 21:07:05 (1763345225) PASS 35 (5s) == sanity-flr test 36a: write to mirrored files ========== 21:07:10 (1763345230) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.407809 s, 30.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (10s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 21:07:20 (1763345240) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339295 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189748 s, 55.3 MB/s PASS 36b (5s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 21:07:25 (1763345245) PASS 36c (6s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 21:07:31 (1763345251) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000688355 s, 1.5 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000821478 s, 1.2 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000685845 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000792501 s, 1.3 MB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 21:07:39 (1763345259) create test files with size 15739118 .. d83631f26b0bbf54320f8424f27ed43c - e4c968250e46e7bf5864c1e6db43974a - 7322bf5d055be3ae97ac6eee99e7be79 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg308-client.virtnet /mnt/lustre (opts:) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre PASS 37 (16s) == sanity-flr test 38: resync ============================ 21:07:55 (1763345275) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (9s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 21:08:05 (1763345285) PASS 39 (5s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 21:08:10 (1763345290) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352448 s, 29.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019182 s, 54.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x41:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (6s) == sanity-flr test 41: lfs mirror resync check =========== 21:08:16 (1763345296) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0431374 s, 48.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.170908 s, 24.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.141024 s, 14.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 21:08:29 (1763345309) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 21:08:32 (1763345312) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 21:08:35 (1763345315) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233218 s, 45.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 21:08:42 (1763345322) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.179526 s, 17.5 MB/s PASS 44a (7s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 21:08:49 (1763345329) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4a:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (6s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 21:08:56 (1763345336) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.180764 s, 58.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg308-client.virtnet /mnt/lustre (opts:) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (7s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 21:09:03 (1763345343) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff958d893aa000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.14746 s, 71.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg308-client.virtnet /mnt/lustre (opts:) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff958d99b6b000.xattr_cache=1 PASS 44e (9s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 21:09:12 (1763345352) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (29s) == sanity-flr test 46: Verify setstripe --copy option ==== 21:09:41 (1763345381) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (29s) == sanity-flr test 47: Verify mirror obj alloc =========== 21:10:10 (1763345410) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 21:10:12 (1763345412) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.224985 s, 14.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.167409 s, 18.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xfbd56e57 (stale) Mirror 2: 0xfbd56e57 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 9834ea5a1a291f51e3375f47fbfbbc8b - mirror 1 checksum: 3667a37d597466199650dda9b6ebc59f - mirror 2 checksum: 9834ea5a1a291f51e3375f47fbfbbc8b - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Nov 16 21:10:15 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Nov 16 21:10:15 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 9834ea5a1a291f51e3375f47fbfbbc8b - mirror 1 checksum: 3667a37d597466199650dda9b6ebc59f - mirror 2 checksum: 3667a37d597466199650dda9b6ebc59f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Nov 16 21:10:15 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Nov 16 21:10:15 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } PASS 48 (8s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 21:10:20 (1763345420) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180335 s, 58.1 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (4s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 21:10:25 (1763345425) Failing mds1 on oleg308-server Stopping /mnt/lustre-mds1 (opts:) on oleg308-server 21:10:34 (1763345434) shut down facet: mds1 facet_host: oleg308-server facet_failover_host: oleg308-server Failover mds1 to oleg308-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0000 21:10:52 (1763345452) targets are mounted 21:10:52 (1763345452) facet_failover done oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201944 s, 51.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } PASS 50A (38s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 21:11:02 (1763345462) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0381163 s, 537 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0133829 s, 1.5 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (7s) == sanity-flr test 50b: mirror rsync handles sparseness == 21:11:09 (1763345469) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0412712 s, 496 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0094517 s, 2.2 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (7s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 21:11:16 (1763345476) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00700716 s, 2.3 MB/s PASS 50c (5s) == sanity-flr test 50d: mirror rsync keep holes ========== 21:11:21 (1763345481) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0576808 s, 355 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (6s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 21:11:27 (1763345487) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00667662 s, 613 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0439602 s, 93.2 kB/s PASS 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 21:11:31 (1763345491) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) PASS 60b (14s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 21:11:45 (1763345505) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (24s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 21:12:09 (1763345529) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 21:12:32 (1763345552) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 21:12:54 (1763345574) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (5s) == sanity-flr test 70: mirror create and split race ====== 21:13:00 (1763345580) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 55937 mirror split pid 55938 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 55937 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 55938 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (65s) == sanity-flr test 70a: flr mode fsx test ================ 21:14:04 (1763345644) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 5645 skipping zero length zero range skipping zero length zero range 000003 1763345645.868250 write 0x008661 thru 0x011124 (0x8ac4 bytes) truncating to largest ever: 0x26d73 000004 1763345645.869782 trunc from 0x011125 to 0x026d72 (0x15c4e bytes) fallocating to largest ever: 0x32714 000005 1763345645.883842 fallocate 0x02ef40 thru 0x032713 (0x37d4 bytes) 000007 1763345645.890949 mapwrite 0x034140 thru 0x0380e2 (0x3fa3 bytes) 000008 1763345645.933757 write 0x03772c thru 0x03ffff (0x88d4 bytes) 000009 1763345645.934977 read 0x03c5f2 thru 0x03ffff (0x3a0e bytes) 000011 1763345645.935871 punch from 0x0313b7 to 0x03ea50 (0xd69a bytes) 000013 1763345645.948867 punch from 0x026486 to 0x027317 (0xe92 bytes) 000014 1763345645.953427 mapwrite 0x00387c thru 0x003d29 (0x4ae bytes) 000015 1763345645.971371 punch from 0x03bf24 to 0x03ffff (0x40dc bytes) 000016 1763345645.977900 write 0x037cf9 thru 0x03ffff (0x8307 bytes) 000017 1763345645.989945 punch from 0x03f614 to 0x03ffff (0x9ec bytes) 000019 1763345646.002515 read 0x00b936 thru 0x00ba5d (0x128 bytes) 000020 1763345646.008945 MIRROR_EXTEND to 2 mirrors 000022 1763345646.511500 mapwrite 0x01e689 thru 0x02e2b1 (0xfc29 bytes) 000024 1763345646.603924 trunc from 0x01046e to 0x017326 (0x6eb9 bytes) 000025 1763345646.609194 punch from 0x02b1de to 0x033cec (0x8b0f bytes) 000026 1763345646.616132 fallocate 0x005c4d thru 0x0085a1 (0x2955 bytes) 000027 1763345646.620121 punch from 0x022745 to 0x02a21d (0x7ad9 bytes) 000028 1763345646.624735 mapread 0x03485e thru 0x03ffff (0xb7a2 bytes) 000029 1763345646.639417 trunc from 0x040000 to 0x03194e (0xe6b1 bytes) 000030 1763345646.669089 write 0x0393b8 thru 0x03ef59 (0x5ba2 bytes) 000031 1763345646.669989 fallocate 0x00489b thru 0x011264 (0xc9ca bytes) 000032 1763345646.674661 mapwrite 0x00a07d thru 0x012cc3 (0x8c47 bytes) zero_range to largest ever: 0x3927d 000033 1763345646.701692 trunc from 0x0338e9 to 0x03927c (0x5994 bytes) 000034 1763345646.708301 mapread 0x039091 thru 0x03ef59 (0x5ec9 bytes) 000035 1763345646.718795 read 0x0331d2 thru 0x03bbdc (0x8a0b bytes) 000036 1763345646.719438 punch from 0x02b31c to 0x02b49c (0x181 bytes) 000038 1763345646.725121 punch from 0x0367f6 to 0x0370ec (0x8f7 bytes) 000039 1763345646.729215 punch from 0x01466c to 0x01d6f3 (0x9088 bytes) 000040 1763345646.733031 fallocate 0x015bd3 thru 0x01b40e (0x583c bytes) 000041 1763345646.737191 fallocate 0x00d70c thru 0x01b417 (0xdd0c bytes) 000042 1763345646.740174 fallocate 0x00dec6 thru 0x00e1a8 (0x2e3 bytes) 000043 1763345646.744730 punch from 0x00f16c to 0x00fe55 (0xcea bytes) 000044 1763345646.750317 punch from 0x0242d8 to 0x02afdf (0x6d08 bytes) 000046 1763345646.760712 MIRROR_RESYNC 2 mirrors 000048 1763345647.228018 punch from 0x00f498 to 0x01648d (0x6ff6 bytes) 000049 1763345647.253715 mapwrite 0x039393 thru 0x039f68 (0xbd6 bytes) 000050 1763345647.276558 trunc from 0x00944a to 0x00a513 (0x10ca bytes) 000051 1763345647.283566 mapread 0x03809f thru 0x039a4f (0x19b1 bytes) 000052 1763345647.289202 fallocate 0x00c194 thru 0x00da62 (0x18cf bytes) 000053 1763345647.296992 mapwrite 0x02905a thru 0x02dcb3 (0x4c5a bytes) 000054 1763345647.318549 punch from 0x00983e to 0x011cac (0x846f bytes) 000055 1763345647.326216 MIRROR_RESYNC 2 mirrors 000056 1763345647.851803 fallocate 0x0296fd thru 0x035a91 (0xc395 bytes) 000057 1763345647.875707 read 0x01d37e thru 0x02a24e (0xced1 bytes) 000058 1763345647.881513 trunc from 0x03ef5a to 0x02b11f (0x13e3a bytes) 000059 1763345647.909457 trunc from 0x02b120 to 0x037a02 (0xc8e3 bytes) 000061 1763345647.929557 trunc from 0x037a03 to 0x037699 (0x369 bytes) 000062 1763345647.955997 mapread 0x034e70 thru 0x037699 (0x282a bytes) 000063 1763345647.976472 MIRROR_RESYNC 2 mirrors 000064 1763345648.481827 read 0x0182a3 thru 0x01b5e2 (0x3340 bytes) 000065 1763345648.487345 write 0x03fafb thru 0x03ffff (0x505 bytes) 000067 1763345648.519577 trunc from 0x040000 to 0x020600 (0x1f9ff bytes) 000068 1763345648.535151 trunc from 0x020601 to 0x03072b (0x1012b bytes) 000069 1763345648.550189 mapread 0x008dad thru 0x00e1e3 (0x5437 bytes) 000071 1763345648.569100 write 0x005030 thru 0x00f120 (0xa0f1 bytes) 000072 1763345648.571138 mapread 0x001f5e thru 0x002c79 (0xd1c bytes) 000073 1763345648.581475 trunc from 0x01aa32 to 0x0290b6 (0xe685 bytes) 000074 1763345648.588006 trunc from 0x0084e3 to 0x00d6a8 (0x51c6 bytes) 000075 1763345648.600881 mapread 0x02d4d6 thru 0x03072b (0x3256 bytes) 000076 1763345648.608902 fallocate 0x02f4b5 thru 0x038307 (0x8e53 bytes) 000077 1763345648.614783 fallocate 0x00cef3 thru 0x00daf1 (0xbff bytes) 000078 1763345648.623368 MIRROR_RESYNC 2 mirrors 000079 1763345649.220138 trunc from 0x038308 to 0x000441 (0x37ec6 bytes) 000080 1763345649.280981 punch from 0x0001a1 to 0x000441 (0x2a1 bytes) 000081 1763345649.293285 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x25c:0x0]: Invalid argument (22) 000082 1763345649.429362 punch from 0x00031c to 0x000441 (0x126 bytes) 000084 1763345649.436625 trunc from 0x000137 to 0x000441 (0x30b bytes) fallocating to largest ever: 0x3c392 000085 1763345649.441304 fallocate 0x0323df thru 0x03c391 (0x9fb3 bytes) 000087 1763345649.446555 write 0x0056ad thru 0x012027 (0xc97b bytes) 000088 1763345649.463008 trunc from 0x010af0 to 0x0168b4 (0x5dc5 bytes) 000089 1763345649.483235 trunc from 0x03c392 to 0x033fab (0x83e6 bytes) 000090 1763345649.494870 trunc from 0x033fac to 0x006145 (0x2de66 bytes) 000091 1763345649.512301 fallocate 0x028819 thru 0x02f916 (0x70fe bytes) 000092 1763345649.521428 mapread 0x003ea3 thru 0x006145 (0x22a3 bytes) 000093 1763345649.527356 trunc from 0x006146 to 0x0068b7 (0x772 bytes) 000094 1763345649.542313 write 0x01659f thru 0x01c580 (0x5fe2 bytes) 000097 1763345649.544427 write 0x014ab3 thru 0x021747 (0xcc95 bytes) 000098 1763345649.563629 read 0x014274 thru 0x01bf13 (0x7ca0 bytes) 000099 1763345649.564129 read 0x00b81a thru 0x0166aa (0xae91 bytes) 000100 1763345649.572106 read 0x009662 thru 0x00c677 (0x3016 bytes) 000101 1763345649.572636 trunc from 0x021748 to 0x00742f (0x1a318 bytes) 000102 1763345649.588544 read 0x001387 thru 0x001a0a (0x684 bytes) 000103 1763345649.589265 mapwrite 0x00810f thru 0x00e590 (0x6482 bytes) 000104 1763345649.625524 read 0x0026fb thru 0x00e590 (0xbe96 bytes) 000105 1763345649.630022 MIRROR_EXTEND to 2 mirrors 000106 1763345650.100351 fallocate 0x001415 thru 0x00f37c (0xdf68 bytes) 000107 1763345650.161795 punch from 0x0000c3 to 0x00b769 (0xb6a7 bytes) 000108 1763345650.181126 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x25c:0x0]: Invalid argument (22) 000109 1763345650.343698 read 0x00e047 thru 0x00e590 (0x54a bytes) 000110 1763345650.353007 write 0x029450 thru 0x02fc36 (0x67e7 bytes) 000111 1763345650.362087 mapread 0x019812 thru 0x01de45 (0x4634 bytes) 000112 1763345650.380955 trunc from 0x01b9a1 to 0x026f7f (0xb5df bytes) 000113 1763345650.385078 trunc from 0x02fc37 to 0x0147df (0x1b457 bytes) 000114 1763345650.429103 write 0x035ea1 thru 0x036a5e (0xbbe bytes) fallocating to largest ever: 0x40000 000115 1763345650.430659 fallocate 0x03830f thru 0x03ffff (0x7cf1 bytes) 000116 1763345650.434454 mapread 0x00fe00 thru 0x01012f (0x330 bytes) 000118 1763345650.443479 mapread 0x00d98f thru 0x01ceba (0xf52c bytes) 000119 1763345650.458807 MIRROR_EXTEND to 2 mirrors 000120 1763345650.956170 fallocate 0x0191ab thru 0x0258db (0xc731 bytes) 000121 1763345650.989960 mapread 0x015523 thru 0x019fe6 (0x4ac4 bytes) 000122 1763345651.005185 punch from 0x0164af to 0x01a51e (0x4070 bytes) 000125 1763345651.009626 read 0x0108cc thru 0x01a8f1 (0xa026 bytes) 000126 1763345651.026090 MIRROR_RESYNC 2 mirrors 000127 1763345651.467862 mapwrite 0x00be1d thru 0x011fc9 (0x61ad bytes) 000128 1763345651.556766 trunc from 0x036a5f to 0x013ca2 (0x22dbc bytes) 000129 1763345651.585549 trunc from 0x00e601 to 0x013ca2 (0x56a2 bytes) 000130 1763345651.591256 punch from 0x0014df to 0x00d4bc (0xbfde bytes) 000131 1763345651.596873 trunc from 0x013ca3 to 0x021c50 (0xdfae bytes) 000133 1763345651.613977 MIRROR_RESYNC 2 mirrors 000134 1763345652.045475 punch from 0x012f9a to 0x01f04b (0xc0b2 bytes) 000135 1763345652.085340 mapwrite 0x03298e thru 0x03db98 (0xb20b bytes) 000136 1763345652.168964 read 0x00f22e thru 0x01871b (0x94ee bytes) 000137 1763345652.186276 trunc from 0x034b11 to 0x034ba1 (0x91 bytes) 000138 1763345652.195535 write 0x03f76e thru 0x03ffff (0x892 bytes) 000139 1763345652.202372 mapread 0x00305a thru 0x01193c (0xe8e3 bytes) 000140 1763345652.224221 write 0x002832 thru 0x0123b8 (0xfb87 bytes) 000141 1763345652.225797 read 0x03a5fb thru 0x03ffff (0x5a05 bytes) 000142 1763345652.238740 read 0x01f326 thru 0x02be4d (0xcb28 bytes) 000143 1763345652.240979 fallocate 0x037945 thru 0x03ffff (0x86bb bytes) 000144 1763345652.256041 fallocate 0x00cbb0 thru 0x013b44 (0x6f95 bytes) 000145 1763345652.265119 punch from 0x005804 to 0x00ce6d (0x766a bytes) 000146 1763345652.299518 fallocate 0x02f086 thru 0x03de61 (0xeddc bytes) 000147 1763345652.312877 write 0x03325e thru 0x03ff58 (0xccfb bytes) 000148 1763345652.314439 fallocate 0x036cea thru 0x03ffff (0x9316 bytes) 000149 1763345652.329555 punch from 0x01a54e to 0x01ba49 (0x14fc bytes) 000150 1763345652.338963 mapread 0x025b0e thru 0x0293d8 (0x38cb bytes) 000151 1763345652.346113 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x25c:0x0]: Invalid argument (22) 000153 1763345652.523103 write 0x015d4f thru 0x019fc3 (0x4275 bytes) 000154 1763345652.538748 punch from 0x028ae8 to 0x02e8d0 (0x5de9 bytes) 000156 1763345652.543549 read 0x024379 thru 0x02507a (0xd02 bytes) 000158 1763345652.550265 punch from 0x02ef31 to 0x035bad (0x6c7d bytes) 000159 1763345652.556094 mapwrite 0x005bff thru 0x00bf44 (0x6346 bytes) 000161 1763345652.580181 MIRROR_EXTEND to 2 mirrors 000162 1763345653.203103 punch from 0x02b14a to 0x030657 (0x550e bytes) 000163 1763345653.237067 read 0x03544f thru 0x03ffff (0xabb1 bytes) 000164 1763345653.244565 trunc from 0x013885 to 0x015f50 (0x26cc bytes) 000165 1763345653.252979 read 0x008068 thru 0x00b3db (0x3374 bytes) 000166 1763345653.270062 mapread 0x031a63 thru 0x03ffff (0xe59d bytes) 000168 1763345653.271442 write 0x026988 thru 0x034aac (0xe125 bytes) 000169 1763345653.279661 mapread 0x022a0f thru 0x025b05 (0x30f7 bytes) 000170 1763345653.286102 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x25c:0x0]: Invalid argument (22) 000171 1763345653.449527 mapwrite 0x02365c thru 0x0329ec (0xf391 bytes) 000172 1763345653.496418 write 0x01dd1d thru 0x02a445 (0xc729 bytes) 000174 1763345653.520312 trunc from 0x03e1c9 to 0x03ffff (0x1e37 bytes) 000176 1763345653.528168 trunc from 0x040000 to 0x001b9c (0x3e463 bytes) 000177 1763345653.575358 mapread 0x001951 thru 0x001b9c (0x24c bytes) 000178 1763345653.580343 fallocate 0x0052e1 thru 0x00ab16 (0x5836 bytes) 000179 1763345653.584834 read 0x0007d1 thru 0x001b9c (0x13cc bytes) 000180 1763345653.589916 mapwrite 0x010208 thru 0x01ac5b (0xaa54 bytes) 000181 1763345653.628499 trunc from 0x016078 to 0x01ac5b (0x4be4 bytes) 000182 1763345653.637491 read 0x011287 thru 0x01ac5b (0x99d5 bytes) 000183 1763345653.649881 mapread 0x00453d thru 0x005a05 (0x14c9 bytes) 000184 1763345653.653039 trunc from 0x01ac5c to 0x00c357 (0xe904 bytes) 000185 1763345653.676534 trunc from 0x003ecb to 0x007a43 (0x3b79 bytes) 000186 1763345653.681707 read 0x0084de thru 0x00c357 (0x3e7a bytes) 000187 1763345653.694574 MIRROR_EXTEND to 2 mirrors 000188 1763345654.183783 punch from 0x000edc to 0x00326a (0x238f bytes) 000190 1763345654.222614 trunc from 0x00c358 to 0x0262d9 (0x19f82 bytes) 000191 1763345654.273972 fallocate 0x00fd08 thru 0x01635c (0x6655 bytes) 000192 1763345654.279441 trunc from 0x011601 to 0x01d2c3 (0xbcc3 bytes) 000193 1763345654.285206 read 0x0061c8 thru 0x012d98 (0xcbd1 bytes) 000194 1763345654.290401 write 0x0396f0 thru 0x03ffff (0x6910 bytes) 000195 1763345654.291538 mapwrite 0x001a23 thru 0x0102c2 (0xe8a0 bytes) 000196 1763345654.327556 fallocate 0x01add0 thru 0x0271bb (0xc3ec bytes) 000197 1763345654.332209 read 0x004936 thru 0x00d328 (0x89f3 bytes) 000198 1763345654.332823 read 0x0269fd thru 0x035df4 (0xf3f8 bytes) 000199 1763345654.341156 mapread 0x0023d5 thru 0x00750a (0x5136 bytes) 000201 1763345654.342384 mapwrite 0x02aec4 thru 0x03947c (0xe5b9 bytes) 000204 1763345654.375468 trunc from 0x040000 to 0x02622b (0x19dd4 bytes) 000205 1763345654.390844 trunc from 0x02622c to 0x0283ec (0x21c1 bytes) 000206 1763345654.406013 fallocate 0x027f48 thru 0x033ca3 (0xbd5c bytes) 000207 1763345654.410899 trunc from 0x0120e3 to 0x01a13a (0x8058 bytes) 000208 1763345654.429018 mapwrite 0x00f8fb thru 0x01e2c7 (0xe9cd bytes) 000209 1763345654.519242 write 0x01bd36 thru 0x025183 (0x944e bytes) 000210 1763345654.520463 punch from 0x020bcc to 0x0279d2 (0x6e07 bytes) 000211 1763345654.540521 fallocate 0x020f98 thru 0x02a79d (0x9806 bytes) 000212 1763345654.550025 punch from 0x01034f to 0x01349b (0x314d bytes) 000213 1763345654.566845 MIRROR_RESYNC 2 mirrors 000214 1763345655.093467 mapread 0x009b14 thru 0x0199da (0xfec7 bytes) 000215 1763345655.123677 punch from 0x0002df to 0x0077c0 (0x74e2 bytes) 000216 1763345655.174128 mapread 0x01b0b1 thru 0x01d9db (0x292b bytes) 000217 1763345655.189493 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2a79e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b000): Mirror 2: 0x9efe8fa8 (stale) CRC-32 checksum value for chunk [0, 0x2b000): Mirror 1: 0x6d78c399 Mirror 3: 0x6d78c399 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b000: skipped Size error: expected 0x2a79e stat 0x2a79e seek 0x2b000 LOG DUMP (186 total operations): 1: 1763345645.864640 SKIPPED (no operation) 2: 1763345645.867849 SKIPPED (no operation) 3: 1763345645.868250 WRITE 0x08661 thru 0x11124 (0x08ac4 bytes) HOLE 4: 1763345645.869782 TRUNC_UP 0x11125 to 0x26d73 5: 1763345645.883842 FALLOC 0x2ef40 thru 0x32714 (0x037d4 bytes)EXTENDING 6: 1763345645.890949 MAPWRITE 0x34140 thru 0x380e2 (0x03fa3 bytes) 7: 1763345645.933757 WRITE 0x3772c thru 0x3ffff (0x088d4 bytes) EXTEND 8: 1763345645.934977 READ 0x3c5f2 thru 0x3ffff (0x03a0e bytes) 9: 1763345645.935871 PUNCH 0x313b7 thru 0x3ea50 (0x0d69a bytes) 10: 1763345645.948867 PUNCH 0x26486 thru 0x27317 (0x00e92 bytes) 11: 1763345645.953427 MAPWRITE 0x0387c thru 0x03d29 (0x004ae bytes) 12: 1763345645.971371 PUNCH 0x3bf24 thru 0x3ffff (0x040dc bytes) 13: 1763345645.977900 WRITE 0x37cf9 thru 0x3ffff (0x08307 bytes) 14: 1763345645.989945 PUNCH 0x3f614 thru 0x3ffff (0x009ec bytes) 15: 1763345646.002515 READ 0x0b936 thru 0x0ba5d (0x00128 bytes) 16: 1763345646.008945 MIRROR_EXTEND to 2 mirrors 17: 1763345646.511500 MAPWRITE 0x1e689 thru 0x2e2b1 (0x0fc29 bytes) 18: 1763345646.603924 ZERO 0x1046e thru 0x17326 (0x06eb9 bytes) 19: 1763345646.609194 PUNCH 0x2b1de thru 0x33cec (0x08b0f bytes) 20: 1763345646.616132 FALLOC 0x05c4d thru 0x085a2 (0x02955 bytes)INTERIOR 21: 1763345646.620121 PUNCH 0x22745 thru 0x2a21d (0x07ad9 bytes) 22: 1763345646.624735 MAPREAD 0x3485e thru 0x3ffff (0x0b7a2 bytes) 23: 1763345646.639417 TRUNC_DN 0x40000 to 0x3194f 24: 1763345646.669089 WRITE 0x393b8 thru 0x3ef59 (0x05ba2 bytes) HOLE 25: 1763345646.669989 FALLOC 0x0489b thru 0x11265 (0x0c9ca bytes)INTERIOR 26: 1763345646.674661 MAPWRITE 0x0a07d thru 0x12cc3 (0x08c47 bytes) 27: 1763345646.701692 ZERO 0x338e9 thru 0x3927c (0x05994 bytes) 28: 1763345646.708301 MAPREAD 0x39091 thru 0x3ef59 (0x05ec9 bytes) 29: 1763345646.718795 READ 0x331d2 thru 0x3bbdc (0x08a0b bytes) 30: 1763345646.719438 PUNCH 0x2b31c thru 0x2b49c (0x00181 bytes) 31: 1763345646.725121 PUNCH 0x367f6 thru 0x370ec (0x008f7 bytes) 32: 1763345646.729215 PUNCH 0x1466c thru 0x1d6f3 (0x09088 bytes) 33: 1763345646.733031 FALLOC 0x15bd3 thru 0x1b40f (0x0583c bytes)INTERIOR 34: 1763345646.737191 FALLOC 0x0d70c thru 0x1b418 (0x0dd0c bytes)INTERIOR 35: 1763345646.740174 FALLOC 0x0dec6 thru 0x0e1a9 (0x002e3 bytes)INTERIOR 36: 1763345646.744730 PUNCH 0x0f16c thru 0x0fe55 (0x00cea bytes) 37: 1763345646.750317 PUNCH 0x242d8 thru 0x2afdf (0x06d08 bytes) 38: 1763345646.760712 MIRROR_RESYNC 2 mirrors 39: 1763345647.228018 PUNCH 0x0f498 thru 0x1648d (0x06ff6 bytes) 40: 1763345647.253715 MAPWRITE 0x39393 thru 0x39f68 (0x00bd6 bytes) 41: 1763345647.276558 ZERO 0x0944a thru 0x0a513 (0x010ca bytes) 42: 1763345647.283566 MAPREAD 0x3809f thru 0x39a4f (0x019b1 bytes) 43: 1763345647.289202 FALLOC 0x0c194 thru 0x0da63 (0x018cf bytes)INTERIOR 44: 1763345647.296992 MAPWRITE 0x2905a thru 0x2dcb3 (0x04c5a bytes) 45: 1763345647.318549 PUNCH 0x0983e thru 0x11cac (0x0846f bytes) 46: 1763345647.326216 MIRROR_RESYNC 2 mirrors 47: 1763345647.851803 FALLOC 0x296fd thru 0x35a92 (0x0c395 bytes)INTERIOR 48: 1763345647.875707 READ 0x1d37e thru 0x2a24e (0x0ced1 bytes) 49: 1763345647.881513 TRUNC_DN 0x3ef5a to 0x2b120 50: 1763345647.909457 TRUNC_UP 0x2b120 to 0x37a03 51: 1763345647.929557 TRUNC_DN 0x37a03 to 0x3769a 52: 1763345647.955997 MAPREAD 0x34e70 thru 0x37699 (0x0282a bytes) 53: 1763345647.976472 MIRROR_RESYNC 2 mirrors 54: 1763345648.481827 READ 0x182a3 thru 0x1b5e2 (0x03340 bytes) 55: 1763345648.487345 WRITE 0x3fafb thru 0x3ffff (0x00505 bytes) HOLE 56: 1763345648.519577 TRUNC_DN 0x40000 to 0x20601 57: 1763345648.535151 TRUNC_UP 0x20601 to 0x3072c 58: 1763345648.550189 MAPREAD 0x08dad thru 0x0e1e3 (0x05437 bytes) 59: 1763345648.569100 WRITE 0x05030 thru 0x0f120 (0x0a0f1 bytes) 60: 1763345648.571138 MAPREAD 0x01f5e thru 0x02c79 (0x00d1c bytes) 61: 1763345648.581475 ZERO 0x1aa32 thru 0x290b6 (0x0e685 bytes) 62: 1763345648.588006 ZERO 0x084e3 thru 0x0d6a8 (0x051c6 bytes) 63: 1763345648.600881 MAPREAD 0x2d4d6 thru 0x3072b (0x03256 bytes) 64: 1763345648.608902 FALLOC 0x2f4b5 thru 0x38308 (0x08e53 bytes)EXTENDING 65: 1763345648.614783 FALLOC 0x0cef3 thru 0x0daf2 (0x00bff bytes)INTERIOR 66: 1763345648.623368 MIRROR_RESYNC 2 mirrors 67: 1763345649.220138 TRUNC_DN 0x38308 to 0x00442 68: 1763345649.280981 PUNCH 0x001a1 thru 0x00441 (0x002a1 bytes) 69: 1763345649.293285 MIRROR_SPLIT mirror 1 to 1 mirrors 70: 1763345649.429362 PUNCH 0x0031c thru 0x00441 (0x00126 bytes) 71: 1763345649.436625 ZERO 0x00137 thru 0x00441 (0x0030b bytes) 72: 1763345649.441304 FALLOC 0x323df thru 0x3c392 (0x09fb3 bytes)EXTENDING 73: 1763345649.446555 WRITE 0x056ad thru 0x12027 (0x0c97b bytes) 74: 1763345649.463008 ZERO 0x10af0 thru 0x168b4 (0x05dc5 bytes) 75: 1763345649.483235 TRUNC_DN 0x3c392 to 0x33fac 76: 1763345649.494870 TRUNC_DN 0x33fac to 0x06146 77: 1763345649.512301 FALLOC 0x28819 thru 0x2f917 (0x070fe bytes)PAST_EOF 78: 1763345649.521428 MAPREAD 0x03ea3 thru 0x06145 (0x022a3 bytes) 79: 1763345649.527356 TRUNC_UP 0x06146 to 0x068b8 80: 1763345649.542313 WRITE 0x1659f thru 0x1c580 (0x05fe2 bytes) HOLE 81: 1763345649.544427 WRITE 0x14ab3 thru 0x21747 (0x0cc95 bytes) EXTEND 82: 1763345649.563629 READ 0x14274 thru 0x1bf13 (0x07ca0 bytes) 83: 1763345649.564129 READ 0x0b81a thru 0x166aa (0x0ae91 bytes) 84: 1763345649.572106 READ 0x09662 thru 0x0c677 (0x03016 bytes) 85: 1763345649.572636 TRUNC_DN 0x21748 to 0x07430 86: 1763345649.588544 READ 0x01387 thru 0x01a0a (0x00684 bytes) 87: 1763345649.589265 MAPWRITE 0x0810f thru 0x0e590 (0x06482 bytes) 88: 1763345649.625524 READ 0x026fb thru 0x0e590 (0x0be96 bytes) 89: 1763345649.630022 MIRROR_EXTEND to 2 mirrors 90: 1763345650.100351 FALLOC 0x01415 thru 0x0f37d (0x0df68 bytes)PAST_EOF 91: 1763345650.161795 PUNCH 0x000c3 thru 0x0b769 (0x0b6a7 bytes) 92: 1763345650.181126 MIRROR_SPLIT mirror 1 to 1 mirrors 93: 1763345650.343698 READ 0x0e047 thru 0x0e590 (0x0054a bytes) 94: 1763345650.353007 WRITE 0x29450 thru 0x2fc36 (0x067e7 bytes) HOLE 95: 1763345650.362087 MAPREAD 0x19812 thru 0x1de45 (0x04634 bytes) 96: 1763345650.380955 ZERO 0x1b9a1 thru 0x26f7f (0x0b5df bytes) 97: 1763345650.385078 TRUNC_DN 0x2fc37 to 0x147e0 98: 1763345650.429103 WRITE 0x35ea1 thru 0x36a5e (0x00bbe bytes) HOLE 99: 1763345650.430659 FALLOC 0x3830f thru 0x40000 (0x07cf1 bytes)PAST_EOF 100: 1763345650.434454 MAPREAD 0x0fe00 thru 0x1012f (0x00330 bytes) 101: 1763345650.443479 MAPREAD 0x0d98f thru 0x1ceba (0x0f52c bytes) 102: 1763345650.458807 MIRROR_EXTEND to 2 mirrors 103: 1763345650.956170 FALLOC 0x191ab thru 0x258dc (0x0c731 bytes)INTERIOR 104: 1763345650.989960 MAPREAD 0x15523 thru 0x19fe6 (0x04ac4 bytes) 105: 1763345651.005185 PUNCH 0x164af thru 0x1a51e (0x04070 bytes) 106: 1763345651.009626 READ 0x108cc thru 0x1a8f1 (0x0a026 bytes) 107: 1763345651.026090 MIRROR_RESYNC 2 mirrors 108: 1763345651.467862 MAPWRITE 0x0be1d thru 0x11fc9 (0x061ad bytes) 109: 1763345651.556766 TRUNC_DN 0x36a5f to 0x13ca3 110: 1763345651.585549 ZERO 0x0e601 thru 0x13ca2 (0x056a2 bytes) 111: 1763345651.591256 PUNCH 0x014df thru 0x0d4bc (0x0bfde bytes) 112: 1763345651.596873 TRUNC_UP 0x13ca3 to 0x21c51 113: 1763345651.613977 MIRROR_RESYNC 2 mirrors 114: 1763345652.045475 PUNCH 0x12f9a thru 0x1f04b (0x0c0b2 bytes) 115: 1763345652.085340 MAPWRITE 0x3298e thru 0x3db98 (0x0b20b bytes) 116: 1763345652.168964 READ 0x0f22e thru 0x1871b (0x094ee bytes) 117: 1763345652.186276 ZERO 0x34b11 thru 0x34ba1 (0x00091 bytes) 118: 1763345652.195535 WRITE 0x3f76e thru 0x3ffff (0x00892 bytes) HOLE 119: 1763345652.202372 MAPREAD 0x0305a thru 0x1193c (0x0e8e3 bytes) 120: 1763345652.224221 WRITE 0x02832 thru 0x123b8 (0x0fb87 bytes) 121: 1763345652.225797 READ 0x3a5fb thru 0x3ffff (0x05a05 bytes) 122: 1763345652.238740 READ 0x1f326 thru 0x2be4d (0x0cb28 bytes) 123: 1763345652.240979 FALLOC 0x37945 thru 0x40000 (0x086bb bytes)INTERIOR 124: 1763345652.256041 FALLOC 0x0cbb0 thru 0x13b45 (0x06f95 bytes)INTERIOR 125: 1763345652.265119 PUNCH 0x05804 thru 0x0ce6d (0x0766a bytes) 126: 1763345652.299518 FALLOC 0x2f086 thru 0x3de62 (0x0eddc bytes)INTERIOR 127: 1763345652.312877 WRITE 0x3325e thru 0x3ff58 (0x0ccfb bytes) 128: 1763345652.314439 FALLOC 0x36cea thru 0x40000 (0x09316 bytes)INTERIOR 129: 1763345652.329555 PUNCH 0x1a54e thru 0x1ba49 (0x014fc bytes) 130: 1763345652.338963 MAPREAD 0x25b0e thru 0x293d8 (0x038cb bytes) 131: 1763345652.346113 MIRROR_SPLIT mirror 1 to 1 mirrors 132: 1763345652.523103 WRITE 0x15d4f thru 0x19fc3 (0x04275 bytes) 133: 1763345652.538748 PUNCH 0x28ae8 thru 0x2e8d0 (0x05de9 bytes) 134: 1763345652.543549 READ 0x24379 thru 0x2507a (0x00d02 bytes) 135: 1763345652.550265 PUNCH 0x2ef31 thru 0x35bad (0x06c7d bytes) 136: 1763345652.556094 MAPWRITE 0x05bff thru 0x0bf44 (0x06346 bytes) 137: 1763345652.580181 MIRROR_EXTEND to 2 mirrors 138: 1763345653.203103 PUNCH 0x2b14a thru 0x30657 (0x0550e bytes) 139: 1763345653.237067 READ 0x3544f thru 0x3ffff (0x0abb1 bytes) 140: 1763345653.244565 ZERO 0x13885 thru 0x15f50 (0x026cc bytes) 141: 1763345653.252979 READ 0x08068 thru 0x0b3db (0x03374 bytes) 142: 1763345653.270062 MAPREAD 0x31a63 thru 0x3ffff (0x0e59d bytes) 143: 1763345653.271442 WRITE 0x26988 thru 0x34aac (0x0e125 bytes) 144: 1763345653.279661 MAPREAD 0x22a0f thru 0x25b05 (0x030f7 bytes) 145: 1763345653.286102 MIRROR_SPLIT mirror 1 to 1 mirrors 146: 1763345653.449527 MAPWRITE 0x2365c thru 0x329ec (0x0f391 bytes) 147: 1763345653.496418 WRITE 0x1dd1d thru 0x2a445 (0x0c729 bytes) 148: 1763345653.520312 ZERO 0x3e1c9 thru 0x3ffff (0x01e37 bytes) 149: 1763345653.528168 TRUNC_DN 0x40000 to 0x01b9d 150: 1763345653.575358 MAPREAD 0x01951 thru 0x01b9c (0x0024c bytes) 151: 1763345653.580343 FALLOC 0x052e1 thru 0x0ab17 (0x05836 bytes)PAST_EOF 152: 1763345653.584834 READ 0x007d1 thru 0x01b9c (0x013cc bytes) 153: 1763345653.589916 MAPWRITE 0x10208 thru 0x1ac5b (0x0aa54 bytes) 154: 1763345653.628499 ZERO 0x16078 thru 0x1ac5b (0x04be4 bytes) 155: 1763345653.637491 READ 0x11287 thru 0x1ac5b (0x099d5 bytes) 156: 1763345653.649881 MAPREAD 0x0453d thru 0x05a05 (0x014c9 bytes) 157: 1763345653.653039 TRUNC_DN 0x1ac5c to 0x0c358 158: 1763345653.676534 ZERO 0x03ecb thru 0x07a43 (0x03b79 bytes) 159: 1763345653.681707 READ 0x084de thru 0x0c357 (0x03e7a bytes) 160: 1763345653.694574 MIRROR_EXTEND to 2 mirrors 161: 1763345654.183783 PUNCH 0x00edc thru 0x0326a (0x0238f bytes) 162: 1763345654.222614 TRUNC_UP 0x0c358 to 0x262da 163: 1763345654.273972 FALLOC 0x0fd08 thru 0x1635d (0x06655 bytes)INTERIOR 164: 1763345654.279441 ZERO 0x11601 thru 0x1d2c3 (0x0bcc3 bytes) 165: 1763345654.285206 READ 0x061c8 thru 0x12d98 (0x0cbd1 bytes) 166: 1763345654.290401 WRITE 0x396f0 thru 0x3ffff (0x06910 bytes) HOLE 167: 1763345654.291538 MAPWRITE 0x01a23 thru 0x102c2 (0x0e8a0 bytes) 168: 1763345654.327556 FALLOC 0x1add0 thru 0x271bc (0x0c3ec bytes)INTERIOR 169: 1763345654.332209 READ 0x04936 thru 0x0d328 (0x089f3 bytes) 170: 1763345654.332823 READ 0x269fd thru 0x35df4 (0x0f3f8 bytes) 171: 1763345654.341156 MAPREAD 0x023d5 thru 0x0750a (0x05136 bytes) 172: 1763345654.342384 MAPWRITE 0x2aec4 thru 0x3947c (0x0e5b9 bytes) 173: 1763345654.375468 TRUNC_DN 0x40000 to 0x2622c 174: 1763345654.390844 TRUNC_UP 0x2622c to 0x283ed 175: 1763345654.406013 FALLOC 0x27f48 thru 0x33ca4 (0x0bd5c bytes)PAST_EOF 176: 1763345654.410899 ZERO 0x120e3 thru 0x1a13a (0x08058 bytes) 177: 1763345654.429018 MAPWRITE 0x0f8fb thru 0x1e2c7 (0x0e9cd bytes) 178: 1763345654.519242 WRITE 0x1bd36 thru 0x25183 (0x0944e bytes) 179: 1763345654.520463 PUNCH 0x20bcc thru 0x279d2 (0x06e07 bytes) 180: 1763345654.540521 FALLOC 0x20f98 thru 0x2a79e (0x09806 bytes)EXTENDING 181: 1763345654.550025 PUNCH 0x1034f thru 0x1349b (0x0314d bytes) 182: 1763345654.566845 MIRROR_RESYNC 2 mirrors 183: 1763345655.093467 MAPREAD 0x09b14 thru 0x199da (0x0fec7 bytes) 184: 1763345655.123677 PUNCH 0x002df thru 0x077c0 (0x074e2 bytes) 185: 1763345655.174128 MAPREAD 0x1b0b1 thru 0x1d9db (0x0292b bytes) 186: 1763345655.189493 MIRROR_EXTEND to 3 mirrors Using seed 5645 Correct content saved for comparison (maybe hexdump "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr" vs "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood") sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7314:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3373:test_70a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7690:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7753:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7568:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3375:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_70a.*.1763345657.log rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.debug_log.oleg308-server.1763345657.log.0j7tVv" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.dmesg.oleg308-server.1763345657.log.uI2dyz" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 70a (17s) == sanity-flr test 71: check mirror extend parent fid ==== 21:14:21 (1763345661) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000404:0x4a:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (8s) == sanity-flr test 200a: stress test ===================== 21:14:29 (1763345669) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre2 Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 8684160 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5390304 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9006496 .. Extending file size to 8153312 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9111872 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9144224 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 59380 59381 59383 59384 59387 192.168.203.108@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre2 (opts:) 192.168.203.108@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (72s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 21:15:41 (1763345741) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre2 Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6711616 .. Extending file size to 3930752 .. Extending file size to 6455360 .. done Extending file size to 7067552 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8006240 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8140992 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8690944 .. Extending file size to 8434240 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9313568 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9143296 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61727 61728 61729 61733 61734 done 192.168.203.108@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre2 (opts:) 192.168.203.108@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (74s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 21:16:55 (1763345815) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2696 1268608 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1860 1269444 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 15048 3591972 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 22980 3584040 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38028 7176012 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2696 1268608 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1860 1269444 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 15048 3591972 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 22980 3584040 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 38028 7176012 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.107136 s, 19.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.139257 s, 15.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xcd3f48d4 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.108@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 21:17:14 (1763345834) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.140303 s, 14.9 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 21:17:20 (1763345840) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x273:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x38e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x38e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x38f:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x38f:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 21:17:27 (1763345847) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131385 s, 16.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x33:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x34:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x35:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 21:17:35 (1763345855) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.120138 s, 17.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x390:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x391:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x390:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x390:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x391:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x390:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x391:0x0] } PASS 204b (6s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 21:17:42 (1763345862) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0945102 s, 22.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x392:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x392:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x392:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x392:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x393:0x0] } PASS 204c (30s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 21:18:12 (1763345892) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg308-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570974 s, 18.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x34:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x37:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x36:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x34:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x37:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x36:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x35:0x0] } PASS 204d (45s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 21:18:57 (1763345937) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg308-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x36:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x37:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0816581 s, 25.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x36:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x37:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x38:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x36:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x37:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x38:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 21:19:43 (1763345983) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg308-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg308-server: Pool lustre.test_204f created Adding targets to pool oleg308-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg308-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0650225 s, 32.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x393:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x394:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x393:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x394:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg308-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg308-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg308-server: Pool lustre.test_204f destroyed PASS 204f (63s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 21:20:46 (1763346046) PASS 205a (5s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 21:20:51 (1763346051) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 21:20:57 (1763346057) oleg308-server: Pool lustre.test_206 created Adding targets to pool oleg308-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg308-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg308-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg308-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg308-server: Pool lustre.test_206 destroyed PASS 206 (24s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 21:21:21 (1763346081) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375541 s, 27.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209047 s, 50.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xfd47f786 Mirror 3: 0xfd47f786 PASS 207 (8s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 21:21:29 (1763346089) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 21:21:32 (1763346092) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 21:21:34 (1763346094) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0578684 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1280000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1630000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1630000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1630000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x17f0000 Read this (4 bytes): BBBB 192.168.203.108@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre2 (opts:) PASS 209a (28s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 21:22:02 (1763346122) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0276682 s, 296 kB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 21:22:07 (1763346127) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345376 s, 30.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x292 lmm_fid: [0x200000405:0x292:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x292 lmm_fid: [0x200000405:0x292:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x39e:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611794 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-16 21:22:09.000000000 -0500 Modify: 2025-11-16 21:22:09.000000000 -0500 Change: 2025-11-16 21:22:10.000000000 -0500 Birth: 2025-11-16 21:22:09.000000000 -0500 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 21:22:14 (1763346134) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274359 s, 38.2 MB/s oleg308-server: debugfs 1.47.3-wc1 (8-Aug-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg308-server: debugfs 1.47.3-wc1 (8-Aug-2025) oleg308-server: O/280000401/d30/39e: File not found by ext2_lookup PASS 210b (17s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 21:22:31 (1763346151) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.129941 s, 315 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0182615 s, 224 kB/s size after second write -rw-r--r-- 1 root root 4096 Nov 16 21:22 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x297:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Nov 16 21:22 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 21:22:38 (1763346158) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 21:22:45 (1763346165) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (7s) == sanity-flr test complete, duration 1329 sec =========== 21:22:52 (1763346172) sanity-flr: FAIL: test_70a fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed === sanity-flr: start cleanup 21:22:53 (1763346173) === === sanity-flr: finish cleanup 21:23:00 (1763346180) ===