-----============= acceptance-small: sanity-flr ============----- Mon Aug 24 15:50:05 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg209-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 15:50:25 (1787601025) === oleg209-client.virtnet: executing check_config_client /mnt/lustre oleg209-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg209-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff98d086593800.idle_timeout=debug osc.lustre-OST0001-osc-ffff98d086593800.idle_timeout=debug disable quota as required oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 15:51:02 (1787601062) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7646] == sanity-flr test 0a: lfs mirror create with -N option == 15:51:04 (1787601064) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:51:23 (1787601083) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:51:26 (1787601086) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 15:51:30 (1787601090) PASS 0d (31s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:52:01 (1787601121) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:52:05 (1787601125) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 15:52:09 (1787601129) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff98d086593800.stats=clear osc.lustre-OST0001-osc-ffff98d086593800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:52:17 (1787601137) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:52:27 (1787601147) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:52:36 (1787601156) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 15:52:39 (1787601159) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 15:52:48 (1787601168) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 15:52:56 (1787601176) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:53:03 (1787601183) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:53:06 (1787601186) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:53:13 (1787601193) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 15:53:24 (1787601204) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:53:30 (1787601210) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.02771 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132008 s, 7.9 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:53:40 (1787601220) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.264311 s, 43.6 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 15:53:48 (1787601228) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:53:58 (1787601238) Stopping /mnt/lustre-ost1 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98d086593800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98d086593800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg209-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98d086593800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98d086593800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg209-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (66s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:55:03 (1787601303) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.278284 s, 33.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98d086593800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98d086593800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg209-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (45s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:55:49 (1787601349) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98d086593800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98d086593800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg209-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98d086593800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98d086593800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg209-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (74s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:57:03 (1787601423) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.46783 s, 11.7 MB/s -rw-r--r-- 1 root root 52428800 Aug 24 15:57 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.100s user 0m0.000s sys 0m0.153s Stopping /mnt/lustre-ost1 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98d086593800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98d086593800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.332s user 0m0.001s sys 0m0.168s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg209-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98d086593800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98d086593800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.119s user 0m0.000s sys 0m0.170s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg209-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg209-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98d086593800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98d086593800.ost_server_uuid in FULL state after 0 sec PASS 33b (95s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:58:38 (1787601518) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:58:41 (1787601521) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:58:44 (1787601524) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 15:58:47 (1787601527) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 15:58:53 (1787601533) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.631434 s, 19.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:59:04 (1787601544) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518553 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023114 s, 45.4 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:59:11 (1787601551) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:59:19 (1787601559) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00462019 s, 222 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00262652 s, 390 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00123164 s, 831 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00113353 s, 903 kB/s flv=10 olv=10 PASS 36d (13s) == sanity-flr test 37: mirror I/O API verification ======= 15:59:32 (1787601572) create test files with size 15745830 .. 225b2f1146e6b8da58d61cf283326810 - 2e666a7beae97e7ef0a185bd4b7e76f0 - a73ff9eebcd4d757beb73540efdfc994 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg209-client.virtnet /mnt/lustre (opts:) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 15:59:56 (1787601596) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 16:00:12 (1787601612) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 16:00:18 (1787601618) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132005 s, 7.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281687 s, 37.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 16:00:27 (1787601627) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.150908 s, 13.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.403136 s, 10.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.171657 s, 12.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 16:00:43 (1787601643) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 16:00:46 (1787601646) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 16:00:49 (1787601649) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378738 s, 27.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 16:00:56 (1787601656) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 16:00:59 (1787601659) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 16:01:08 (1787601668) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.375078 s, 28.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg209-client.virtnet /mnt/lustre (opts:) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 16:01:21 (1787601681) llite.lustre-ffff98d089ce4800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.152256 s, 68.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg209-client.virtnet /mnt/lustre (opts:) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff98d08a678000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 16:01:34 (1787601694) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (39s) == sanity-flr test 46: Verify setstripe --copy option ==== 16:02:13 (1787601733) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (33s) == sanity-flr test 47: Verify mirror obj alloc =========== 16:02:46 (1787601766) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 16:02:50 (1787601770) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.348208 s, 9.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.336743 s, 9.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xed3f42ce (nosync,stale) Mirror 2: 0xed3f42ce (nosync,stale) ** verify mirror 2 doesn't change original checksum: cd0d095786098298ef12d7437d378ddf - mirror 1 checksum: add21cb0eb82ab5ac703873db0730c30 - mirror 2 checksum: cd0d095786098298ef12d7437d378ddf - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-24T16:02:53-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-24T16:02:53-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: cd0d095786098298ef12d7437d378ddf - mirror 1 checksum: add21cb0eb82ab5ac703873db0730c30 - mirror 2 checksum: add21cb0eb82ab5ac703873db0730c30 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-24T16:02:53-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-24T16:02:53-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 16:03:02 (1787601782) Failing mds1 on oleg209-server Stopping /mnt/lustre-mds1 (opts:) on oleg209-server 16:03:08 (1787601788) shut down facet: mds1 facet_host: oleg209-server facet_failover_host: oleg209-server Failover mds1 to oleg209-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0000 16:03:30 (1787601810) targets are mounted 16:03:30 (1787601810) facet_failover done oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244661 s, 42.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } PASS 50A (43s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 16:03:45 (1787601825) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 16:03:48 (1787601828) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 16:03:51 (1787601831) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0105331 s, 1.6 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 16:03:58 (1787601838) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 16:04:01 (1787601841) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00846946 s, 484 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0382855 s, 107 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 16:04:08 (1787601848) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 16:04:29 (1787601869) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 16:04:57 (1787601897) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 16:05:21 (1787601921) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 16:05:46 (1787601946) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 16:05:53 (1787601953) mirror create pid 54343 mirror split pid 54344 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54343 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54344 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 16:07:00 (1787602020) Seed set to 2023 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000004 1787602023.447664 mapwrite 0x01e996 thru 0x0231af (0x481a bytes) truncating to largest ever: 0x1d1ef 000005 1787602023.708764 trunc from 0x0231b0 to 0x01d1ee (0x5fc1 bytes) 000010 1787602023.760626 write 0x015da2 thru 0x01e4a7 (0x8706 bytes) 000011 1787602023.797386 read 0x0153e8 thru 0x0160a7 (0xcc0 bytes) 000014 1787602023.799001 read 0x01ddbf thru 0x01e4a7 (0x6e9 bytes) 000015 1787602023.799497 mapwrite 0x03bf93 thru 0x03ceda (0xf48 bytes) 000016 1787602023.937441 trunc from 0x03cedb to 0x006470 (0x36a6a bytes) 000017 1787602023.979497 MIRROR_EXTEND to 2 mirrors 000018 1787602024.965984 mapread 0x00027f thru 0x003d3f (0x3ac1 bytes) 000019 1787602024.993067 write 0x00c4bf thru 0x014537 (0x8079 bytes) truncating to largest ever: 0x2fbcd 000021 1787602025.045461 trunc from 0x014538 to 0x02fbcc (0x1b695 bytes) 000031 1787602025.109202 MIRROR_RESYNC 2 mirrors 000032 1787602025.845499 trunc from 0x02fbcd to 0x0225e2 (0xd5ea bytes) 000034 1787602025.917365 mapread 0x007a5a thru 0x00e18f (0x6736 bytes) 000035 1787602025.940985 trunc from 0x0225e3 to 0x0100ed (0x124f5 bytes) 000036 1787602025.969914 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x100ee): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x100ee): Mirror 2: 0x6dbc730d (stale) CRC-32 checksum value for chunk [0, 0x100ee): Mirror 1: 0x6dbc730d Mirror 3: 0x6dbc730d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x100ee: skipped 000040 1787602027.036438 read 0x009e0c thru 0x00ad9f (0xf94 bytes) 000042 1787602027.042190 read 0x003281 thru 0x005d6a (0x2aea bytes) 000044 1787602027.052453 write 0x01ebf2 thru 0x024f0c (0x631b bytes) 000047 1787602027.188806 write 0x00ed68 thru 0x01d1af (0xe448 bytes) 000048 1787602027.200384 write 0x01a116 thru 0x01af32 (0xe1d bytes) 000049 1787602027.201024 read 0x01cba1 thru 0x024f0c (0x836c bytes) 000050 1787602027.201582 trunc from 0x024f0d to 0x01db9a (0x7372 bytes) 000052 1787602027.253073 read 0x00053d thru 0x006cf6 (0x67ba bytes) 000054 1787602027.265398 mapwrite 0x03b352 thru 0x03ffff (0x4cae bytes) 000055 1787602027.381275 trunc from 0x040000 to 0x01616c (0x29e93 bytes) 000056 1787602027.413496 mapread 0x015338 thru 0x01616c (0xe35 bytes) 000058 1787602027.418300 trunc from 0x01616d to 0x00ec26 (0x7546 bytes) 000059 1787602027.445296 write 0x012fa2 thru 0x01956b (0x65ca bytes) 000062 1787602027.447471 mapwrite 0x03047c thru 0x035b52 (0x56d7 bytes) 000063 1787602027.650495 read 0x01d43f thru 0x01d7c1 (0x383 bytes) 000064 1787602027.675368 write 0x03418f thru 0x03ac79 (0x6aeb bytes) 000065 1787602027.684299 MIRROR_RESYNC 3 mirrors 000067 1787602028.690322 mapwrite 0x039527 thru 0x03ffff (0x6ad9 bytes) 000071 1787602028.817915 write 0x015627 thru 0x01c5ae (0x6f88 bytes) 000072 1787602028.828042 mapread 0x001d82 thru 0x010301 (0xe580 bytes) 000075 1787602028.842295 write 0x03b7e0 thru 0x03ffff (0x4820 bytes) 000078 1787602028.844519 mapread 0x01ba72 thru 0x028272 (0xc801 bytes) 000080 1787602028.875552 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1087d55e (stale) Mirror 3: 0x1087d55e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9f2f88ed Mirror 4: 0x9f2f88ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000082 1787602029.927399 read 0x03b047 thru 0x03ffff (0x4fb9 bytes) 000084 1787602029.933493 mapread 0x019433 thru 0x0255b5 (0xc183 bytes) 000087 1787602029.959360 mapwrite 0x037186 thru 0x03ffff (0x8e7a bytes) 000088 1787602030.119027 mapwrite 0x01a991 thru 0x024332 (0x99a2 bytes) truncating to largest ever: 0x364db 000089 1787602030.189899 trunc from 0x040000 to 0x0364da (0x9b25 bytes) 000091 1787602030.208055 read 0x027d26 thru 0x02c53e (0x4819 bytes) 000095 1787602030.222314 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x364db): Mirror 3: 0x80d9fd0f (stale) Mirror 4: 0x9ff6c951 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x364db): Mirror 3: 0x80d9fd0f (stale) Mirror 4: 0x9ff6c951 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x364db: skipped 000096 1787602030.512242 mapread 0x00dc73 thru 0x00fd74 (0x2102 bytes) 000098 1787602030.533699 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x364db): Mirror 4: 0x9ff6c951 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x364db): Mirror 4: 0x9ff6c951 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x364db: skipped 000099 1787602030.777469 write 0x039668 thru 0x03ffff (0x6998 bytes) 000100 1787602030.778654 mapwrite 0x008594 thru 0x015a3f (0xd4ac bytes) 000101 1787602030.844771 mapread 0x014404 thru 0x01c872 (0x846f bytes) 000104 1787602030.855917 write 0x01a779 thru 0x0266b4 (0xbf3c bytes) 000107 1787602030.867877 read 0x016469 thru 0x022125 (0xbcbd bytes) 000109 1787602030.869098 trunc from 0x040000 to 0x00e2c2 (0x31d3d bytes) 000110 1787602030.890596 trunc from 0x00e2c3 to 0x01ecfd (0x10a3b bytes) 000114 1787602030.916387 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) 000116 1787602031.034506 mapread 0x00a45f thru 0x018630 (0xe1d2 bytes) 000117 1787602031.069647 mapwrite 0x01f5f8 thru 0x0278b8 (0x82c1 bytes) 000121 1787602031.139784 write 0x01ea8d thru 0x029efb (0xb46f bytes) 000122 1787602031.142019 mapread 0x025425 thru 0x029efb (0x4ad7 bytes) 000124 1787602031.143936 mapread 0x00f9ee thru 0x0130c0 (0x36d3 bytes) 000125 1787602031.145067 mapwrite 0x00dcb0 thru 0x018b5a (0xaeab bytes) 000126 1787602031.189288 trunc from 0x029efc to 0x01bcbe (0xe23d bytes) 000132 1787602031.207151 read 0x002cd3 thru 0x007db3 (0x50e1 bytes) 000134 1787602031.213898 read 0x007691 thru 0x014ee5 (0xd855 bytes) 000138 1787602031.216394 mapwrite 0x02a107 thru 0x030696 (0x6590 bytes) 000140 1787602031.282297 write 0x00f1e7 thru 0x01beb9 (0xccd3 bytes) 000146 1787602031.288089 write 0x03dbd6 thru 0x03ebf9 (0x1024 bytes) 000147 1787602031.289359 write 0x0191b5 thru 0x0265b3 (0xd3ff bytes) 000148 1787602031.291352 write 0x015e8e thru 0x019f4b (0x40be bytes) 000151 1787602031.293649 trunc from 0x03ebfa to 0x0034b3 (0x3b746 bytes) 000152 1787602031.308030 mapread 0x0000b2 thru 0x0034b3 (0x3402 bytes) 000153 1787602031.308913 mapread 0x0018f2 thru 0x0034b3 (0x1bc2 bytes) 000154 1787602031.309593 write 0x00c8b5 thru 0x0115a4 (0x4cf0 bytes) 000158 1787602031.317277 trunc from 0x0115a5 to 0x008442 (0x9162 bytes) 000160 1787602031.333380 trunc from 0x008443 to 0x02a02e (0x21bec bytes) 000161 1787602031.354968 read 0x01decc thru 0x025d13 (0x7e48 bytes) 000164 1787602031.368389 mapread 0x003e9a thru 0x00d0c1 (0x9228 bytes) 000167 1787602031.371699 read 0x027715 thru 0x02a02e (0x291a bytes) 000169 1787602031.373077 mapwrite 0x00e0b4 thru 0x013045 (0x4f92 bytes) 000170 1787602031.415610 read 0x018856 thru 0x01d7e0 (0x4f8b bytes) 000172 1787602031.417374 read 0x028c9e thru 0x02a02e (0x1391 bytes) 000174 1787602031.418905 read 0x01edb9 thru 0x02a02e (0xb276 bytes) 000176 1787602031.422362 read 0x0157d2 thru 0x025250 (0xfa7f bytes) 000177 1787602031.423445 mapwrite 0x009f69 thru 0x00a65d (0x6f5 bytes) 000180 1787602031.458259 write 0x01d1dc thru 0x01efd4 (0x1df9 bytes) 000188 1787602031.464173 mapwrite 0x008ba9 thru 0x010bab (0x8003 bytes) 000189 1787602031.507555 mapwrite 0x0118d0 thru 0x01b952 (0xa083 bytes) 000190 1787602031.555433 mapwrite 0x005de2 thru 0x00b3dd (0x55fc bytes) 000191 1787602031.595442 read 0x023273 thru 0x02a02e (0x6dbc bytes) 000194 1787602031.597570 trunc from 0x02a02f to 0x003630 (0x269fe bytes) 000195 1787602031.614637 mapread 0x00080e thru 0x003630 (0x2e23 bytes) 000197 1787602031.619113 write 0x021124 thru 0x028cf2 (0x7bcf bytes) 000200 1787602031.621757 trunc from 0x028cf3 to 0x022b08 (0x61ea bytes) 000201 1787602031.638766 mapread 0x018c8f thru 0x021e39 (0x91ab bytes) 000202 1787602031.649674 write 0x01a51f thru 0x024fe1 (0xaac3 bytes) 000203 1787602031.651933 mapwrite 0x00c718 thru 0x01c398 (0xfc81 bytes) 000204 1787602031.697314 write 0x037015 thru 0x03e8b6 (0x78a2 bytes) 000205 1787602031.699464 write 0x02cd48 thru 0x0372bf (0xa578 bytes) 000209 1787602031.710966 mapwrite 0x01da25 thru 0x029bc2 (0xc19e bytes) 000210 1787602031.756334 mapwrite 0x03b08f thru 0x03d725 (0x2697 bytes) 000211 1787602031.791422 write 0x032a2d thru 0x03bb19 (0x90ed bytes) 000215 1787602031.794824 mapwrite 0x020aa5 thru 0x026042 (0x559e bytes) 000216 1787602031.829726 mapread 0x035ff7 thru 0x03e3a5 (0x83af bytes) 000220 1787602031.831989 mapread 0x037885 thru 0x03e8b6 (0x7032 bytes) 000221 1787602031.833693 mapread 0x0096a3 thru 0x012066 (0x89c4 bytes) 000222 1787602031.835052 write 0x03f3a0 thru 0x03ffff (0xc60 bytes) 000225 1787602031.846192 write 0x02306b thru 0x02f758 (0xc6ee bytes) 000226 1787602031.850009 mapread 0x0095e7 thru 0x00d4d4 (0x3eee bytes) 000227 1787602031.851216 mapwrite 0x030990 thru 0x0356b0 (0x4d21 bytes) 000230 1787602031.893891 trunc from 0x040000 to 0x024cb1 (0x1b34e bytes) 000234 1787602031.914246 mapread 0x01575b thru 0x023d3f (0xe5e5 bytes) 000235 1787602031.915285 read 0x0017d7 thru 0x00388f (0x20b9 bytes) 000237 1787602031.917111 trunc from 0x024cb2 to 0x01d284 (0x7a2d bytes) 000238 1787602031.931559 mapread 0x0099e7 thru 0x00efd6 (0x55f0 bytes) 000239 1787602031.933245 read 0x00c03b thru 0x012333 (0x62f9 bytes) 000240 1787602031.934520 read 0x00249d thru 0x009496 (0x6ffa bytes) truncating to largest ever: 0x3da50 000241 1787602031.935647 trunc from 0x01d285 to 0x03da4f (0x207cb bytes) 000242 1787602031.951958 write 0x017efd thru 0x024b1c (0xcc20 bytes) 000246 1787602031.960491 mapwrite 0x02b231 thru 0x033b20 (0x88f0 bytes) 000247 1787602032.012508 mapread 0x02a93f thru 0x030f0c (0x65ce bytes) 000248 1787602032.013659 trunc from 0x03da50 to 0x03a58a (0x34c5 bytes) 000250 1787602032.030950 write 0x016067 thru 0x02364f (0xd5e9 bytes) 000252 1787602032.033680 trunc from 0x03a58b to 0x0093f3 (0x31197 bytes) 000253 1787602032.051208 MIRROR_EXTEND to 2 mirrors 000257 1787602032.556257 write 0x00d11b thru 0x013441 (0x6327 bytes) 000258 1787602032.642479 read 0x006d1c thru 0x013160 (0xc445 bytes) 000259 1787602032.655118 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x13442): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13442): Mirror 2: 0x4d3097a2 (stale) CRC-32 checksum value for chunk [0, 0x13442): Mirror 1: 0x6f4aeb88 Mirror 3: 0x6f4aeb88 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13442: skipped 000261 1787602033.586259 trunc from 0x013442 to 0x0106a8 (0x2d99 bytes) 000263 1787602033.656073 mapwrite 0x02e656 thru 0x0380f4 (0x9a9f bytes) 000264 1787602033.740729 trunc from 0x0380f5 to 0x031019 (0x70db bytes) 000265 1787602033.759711 mapread 0x01c77e thru 0x029ece (0xd751 bytes) 000266 1787602033.774439 trunc from 0x03101a to 0x02389e (0xd77b bytes) 000267 1787602033.792381 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2389f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2389f): Mirror 2: 0xb74773c0 (stale) Mirror 3: 0x180c1fdf (stale) CRC-32 checksum value for chunk [0, 0x2389f): Mirror 1: 0x5e698266 Mirror 4: 0x5e698266 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2389f: skipped 000269 1787602034.773622 mapwrite 0x03bcfc thru 0x03db71 (0x1e76 bytes) 000271 1787602034.973391 trunc from 0x03db72 to 0x02a61f (0x13552 bytes) 000272 1787602035.008002 mapread 0x022755 thru 0x02a61f (0x7ecb bytes) 000274 1787602035.017153 mapread 0x00b2c6 thru 0x01061b (0x5356 bytes) 000275 1787602035.028419 mapwrite 0x01ac33 thru 0x0283b6 (0xd784 bytes) 000276 1787602035.116106 write 0x03e4b9 thru 0x03ffff (0x1b47 bytes) 000277 1787602035.123588 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0x9367346d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0x9367346d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000279 1787602035.464831 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0x9367346d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6a29a829 Mirror 4: 0x6a29a829 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000281 1787602036.866180 read 0x000c57 thru 0x007dd0 (0x717a bytes) 000283 1787602036.884457 write 0x023356 thru 0x031242 (0xdeed bytes) 000285 1787602037.010982 mapwrite 0x02ef38 thru 0x035c05 (0x6cce bytes) 000287 1787602037.101089 mapread 0x00c79c thru 0x017240 (0xaaa5 bytes) 000288 1787602037.117979 mapread 0x020ada thru 0x0244b1 (0x39d8 bytes) 000293 1787602037.131631 trunc from 0x040000 to 0x03d7e9 (0x2816 bytes) 000296 1787602037.198084 mapread 0x003f1b thru 0x007ee1 (0x3fc7 bytes) 000298 1787602037.215583 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3d7ea): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d7ea): Mirror 2: 0x10f91011 (stale) Mirror 3: 0xf4da16b7 (stale) Mirror 4: 0x44b104ee (stale) CRC-32 checksum value for chunk [0, 0x3d7ea): Mirror 1: 0x29398f80 Mirror 5: 0x29398f80 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d7ea: skipped 000300 1787602038.221480 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3d7ea): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d7ea): Mirror 2: 0x10f91011 (stale) Mirror 3: 0xf4da16b7 (stale) Mirror 4: 0x44b104ee (stale) CRC-32 checksum value for chunk [0, 0x3d7ea): Mirror 1: 0x29398f80 Mirror 5: 0x29398f80 Mirror 6: 0x29398f80 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d7ea: skipped 000302 1787602039.076766 read 0x020b23 thru 0x027526 (0x6a04 bytes) 000305 1787602039.085357 read 0x035d0a thru 0x03d7e9 (0x7ae0 bytes) 000310 1787602039.101073 MIRROR_RESYNC 6 mirrors 000311 1787602040.207084 trunc from 0x03d7ea to 0x031be6 (0xbc03 bytes) 000312 1787602040.343009 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31be7): Mirror 2: 0xe2f70e32 (stale) Mirror 3: 0xe2f70e32 (stale) Mirror 5: 0xe2f70e32 (stale) Mirror 6: 0xe2f70e32 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31be7): Mirror 2: 0xe2f70e32 (stale) Mirror 3: 0xe2f70e32 (stale) Mirror 5: 0xe2f70e32 (stale) Mirror 6: 0xe2f70e32 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31be7: skipped 000313 1787602040.774294 MIRROR_RESYNC 5 mirrors 000317 1787602041.889057 write 0x027921 thru 0x02ac90 (0x3370 bytes) 000319 1787602042.056776 MIRROR_RESYNC 5 mirrors truncating to largest ever: 0x3f7a6 000320 1787602043.747099 trunc from 0x031be7 to 0x03f7a5 (0xdbbf bytes) 000321 1787602043.953343 mapread 0x037075 thru 0x039520 (0x24ac bytes) 000323 1787602043.976472 trunc from 0x03f7a6 to 0x00c75c (0x33049 bytes) 000324 1787602044.006398 mapread 0x00b67d thru 0x00c75c (0x10e0 bytes) 000325 1787602044.019666 trunc from 0x00c75d to 0x0209d1 (0x14275 bytes) 000327 1787602044.046817 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x209d2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x209d2): Mirror 2: 0x17c041e7 (stale) Mirror 3: 0x17c041e7 (stale) Mirror 5: 0x17c041e7 (stale) Mirror 6: 0x17c041e7 (stale) CRC-32 checksum value for chunk [0, 0x209d2): Mirror 1: 0xb67af693 Mirror 7: 0xb67af693 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x209d2: skipped 000330 1787602045.254425 write 0x01c060 thru 0x01d0c6 (0x1067 bytes) 000332 1787602045.418443 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x209d2): Mirror 3: 0x17c041e7 (stale) Mirror 5: 0x17c041e7 (stale) Mirror 6: 0x17c041e7 (stale) Mirror 7: 0xb67af693 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x209d2): Mirror 3: 0x17c041e7 (stale) Mirror 5: 0x17c041e7 (stale) Mirror 6: 0x17c041e7 (stale) Mirror 7: 0xb67af693 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x209d2: skipped 000333 1787602045.893213 mapwrite 0x0033c2 thru 0x004237 (0xe76 bytes) 000334 1787602045.967173 trunc from 0x0209d2 to 0x00748b (0x19546 bytes) 000336 1787602045.999244 mapread 0x000cde thru 0x00748b (0x67ae bytes) 000338 1787602046.051148 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x748c): Mirror 5: 0x104d4cc6 (stale) Mirror 6: 0x104d4cc6 (stale) Mirror 7: 0x104d4cc6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x748c): Mirror 5: 0x104d4cc6 (stale) Mirror 6: 0x104d4cc6 (stale) Mirror 7: 0x104d4cc6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x748c: skipped 000339 1787602046.559444 mapread 0x00375a thru 0x0060dc (0x2983 bytes) 000340 1787602046.621958 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x748c): Mirror 5: 0x104d4cc6 (stale) Mirror 6: 0x104d4cc6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x748c): Mirror 5: 0x104d4cc6 (stale) Mirror 6: 0x104d4cc6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x748c: skipped 000342 1787602046.913712 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x748c): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x748c): Mirror 5: 0x104d4cc6 (stale) Mirror 6: 0x104d4cc6 (stale) CRC-32 checksum value for chunk [0, 0x748c): Mirror 1: 0xbb246a05 Mirror 7: 0xbb246a05 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x748c: skipped 000344 1787602047.800385 read 0x003876 thru 0x00748b (0x3c16 bytes) 000345 1787602047.808472 read 0x00081d thru 0x00748b (0x6c6f bytes) 000346 1787602047.822539 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x748c): Mirror 5: 0x104d4cc6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x748c): Mirror 5: 0x104d4cc6 (stale) CRC-32 checksum value for chunk [0, 0x748c): Mirror 1: 0xbb246a05 Mirror 7: 0xbb246a05 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x748c: skipped 000347 1787602048.209636 read 0x007298 thru 0x00748b (0x1f4 bytes) 000349 1787602048.217127 read 0x00280e thru 0x0060fd (0x38f0 bytes) 000353 1787602048.231228 write 0x01888d thru 0x01cc4b (0x43bf bytes) 000357 1787602048.310622 trunc from 0x01cc4c to 0x00dbbe (0xf08d bytes) 000359 1787602048.345225 read 0x006766 thru 0x00dbbe (0x7459 bytes) 000360 1787602048.355439 write 0x030391 thru 0x03c98b (0xc5fb bytes) 000361 1787602048.356902 write 0x00823a thru 0x00a0d3 (0x1e9a bytes) 000362 1787602048.363034 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3c98c): Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c98c): Mirror 5: 0xadcd6048 (stale) Mirror 7: 0xcd937fac (stale) CRC-32 checksum value for chunk [0, 0x3c98c): Mirror 1: 0x8ab9651c Mirror 8: 0x8ab9651c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c98c: skipped 000363 1787602049.105677 mapread 0x029ce7 thru 0x02b1c2 (0x14dc bytes) 000366 1787602049.125048 trunc from 0x03c98c to 0x034186 (0x8805 bytes) 000368 1787602049.221458 write 0x03f7f1 thru 0x03ffff (0x80f bytes) 000369 1787602049.226024 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xef63959d (stale) Mirror 7: 0x9894ce86 (stale) Mirror 8: 0x28d3ad42 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf5160a09 Mirror 9: 0xf5160a09 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000370 1787602050.188252 mapwrite 0x00ef9a thru 0x013cc2 (0x4d29 bytes) 000371 1787602050.393134 MIRROR_RESYNC 5 mirrors 000376 1787602051.480669 read 0x00e61b thru 0x01e560 (0xff46 bytes) 000377 1787602051.492010 MIRROR_RESYNC 5 mirrors 000379 1787602051.879428 mapread 0x00001f thru 0x00ad50 (0xad32 bytes) 000380 1787602051.903988 mapread 0x03895f thru 0x03ffff (0x76a1 bytes) 000381 1787602051.913276 mapread 0x03ebdd thru 0x03ffff (0x1423 bytes) 000383 1787602051.914187 mapwrite 0x0315ce thru 0x03ef68 (0xd99b bytes) 000385 1787602052.060019 mapwrite 0x02926a thru 0x0346bd (0xb454 bytes) 000387 1787602052.111599 mapread 0x029203 thru 0x02dc2b (0x4a29 bytes) 000389 1787602052.112961 trunc from 0x040000 to 0x02c1c1 (0x13e3e bytes) 000392 1787602052.139723 mapwrite 0x038e41 thru 0x03ffff (0x71bf bytes) 000397 1787602052.210103 MIRROR_RESYNC 5 mirrors 000399 1787602053.759693 write 0x039483 thru 0x03bcb1 (0x282f bytes) 000400 1787602053.840476 mapwrite 0x00b6d9 thru 0x00e138 (0x2a60 bytes) 000402 1787602053.915986 MIRROR_RESYNC 5 mirrors 000403 1787602055.385261 MIRROR_EXTEND to 6 mirrors 000404 1787602055.952129 write 0x00f203 thru 0x019bcf (0xa9cd bytes) 000407 1787602056.036132 write 0x032e1e thru 0x038e4d (0x6030 bytes) 000408 1787602056.045089 read 0x030da1 thru 0x03d4d4 (0xc734 bytes) 000412 1787602056.055993 mapwrite 0x02d504 thru 0x0325b8 (0x50b5 bytes) 000413 1787602056.112166 write 0x0115a1 thru 0x01ea58 (0xd4b8 bytes) 000414 1787602056.122241 MIRROR_RESYNC 6 mirrors 000416 1787602057.421016 read 0x00f457 thru 0x01e9e7 (0xf591 bytes) 000417 1787602057.435606 MIRROR_RESYNC 6 mirrors 000418 1787602057.926627 mapwrite 0x0227b0 thru 0x025121 (0x2972 bytes) 000419 1787602058.125115 read 0x026ade thru 0x0315b2 (0xaad5 bytes) 000420 1787602058.137383 read 0x01fa42 thru 0x02f2df (0xf89e bytes) 000422 1787602058.138465 write 0x02d8f9 thru 0x038f78 (0xb680 bytes) 000424 1787602058.156058 read 0x006433 thru 0x00947c (0x304a bytes) 000425 1787602058.157139 mapread 0x01c7b7 thru 0x01ffce (0x3818 bytes) 000426 1787602058.165034 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x3af2c1a4 (stale) Mirror 8: 0x3af2c1a4 (stale) Mirror 9: 0x3af2c1a4 (stale) Mirror 10: 0x3af2c1a4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x3af2c1a4 (stale) Mirror 8: 0x3af2c1a4 (stale) Mirror 9: 0x3af2c1a4 (stale) Mirror 10: 0x3af2c1a4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000428 1787602058.610615 read 0x03ad22 thru 0x03f2c7 (0x45a6 bytes) 000430 1787602058.625373 trunc from 0x040000 to 0x03bbe2 (0x441d bytes) 000431 1787602058.642395 mapwrite 0x00b4c0 thru 0x0133a9 (0x7eea bytes) truncating to largest ever: 0x3faae 000434 1787602058.709288 trunc from 0x03bbe3 to 0x03faad (0x3ecb bytes) 000435 1787602058.721176 mapread 0x01b504 thru 0x0260ab (0xaba8 bytes) 000437 1787602058.745014 write 0x014183 thru 0x01bfe6 (0x7e64 bytes) 000438 1787602058.747767 trunc from 0x03faae to 0x036e02 (0x8cab bytes) 000440 1787602058.771967 write 0x028c6a thru 0x02f942 (0x6cd9 bytes) 000441 1787602058.781689 write 0x037a06 thru 0x03ffff (0x85fa bytes) 000442 1787602058.789038 trunc from 0x040000 to 0x01c236 (0x23dc9 bytes) 000443 1787602058.806878 mapwrite 0x032333 thru 0x0395e4 (0x72b2 bytes) 000444 1787602058.888727 trunc from 0x0395e5 to 0x012858 (0x26d8c bytes) 000445 1787602058.912786 read 0x0116b1 thru 0x012858 (0x11a8 bytes) 000448 1787602058.915012 trunc from 0x012859 to 0x01ad21 (0x84c9 bytes) 000449 1787602058.931189 write 0x0086fb thru 0x011045 (0x894b bytes) 000450 1787602058.934055 trunc from 0x01ad22 to 0x024f1b (0xa1fa bytes) 000453 1787602058.946228 mapwrite 0x0079f4 thru 0x010c2f (0x923c bytes) 000454 1787602058.990026 read 0x002a85 thru 0x00341f (0x99b bytes) 000455 1787602059.058605 MIRROR_RESYNC 5 mirrors 000456 1787602060.086215 mapread 0x00fcbb thru 0x0137b9 (0x3aff bytes) 000459 1787602060.105137 write 0x032614 thru 0x037590 (0x4f7d bytes) 000460 1787602060.159339 write 0x02669f thru 0x030486 (0x9de8 bytes) 000463 1787602060.175814 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37591): Mirror 8: 0x6c36357 (stale) Mirror 9: 0x6c36357 (stale) Mirror 10: 0x6c36357 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37591): Mirror 8: 0x6c36357 (stale) Mirror 9: 0x6c36357 (stale) Mirror 10: 0x6c36357 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37591: skipped 000467 1787602060.488826 trunc from 0x037591 to 0x03ee9f (0x790f bytes) 000468 1787602060.503390 mapwrite 0x017650 thru 0x023d7d (0xc72e bytes) 000469 1787602060.566927 mapread 0x00383c thru 0x00b079 (0x783e bytes) 000472 1787602060.577968 write 0x00d6ea thru 0x017cb4 (0xa5cb bytes) 000474 1787602060.580391 write 0x03a72b thru 0x03ffff (0x58d5 bytes) 000478 1787602060.589433 mapwrite 0x01ff5e thru 0x023cbe (0x3d61 bytes) 000480 1787602060.639289 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe00e18c3 (stale) Mirror 10: 0xe00e18c3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe00e18c3 (stale) Mirror 10: 0xe00e18c3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000481 1787602060.910168 mapread 0x00ff5b thru 0x013d20 (0x3dc6 bytes) 000482 1787602060.927745 trunc from 0x040000 to 0x012f42 (0x2d0bd bytes) 000484 1787602060.943664 write 0x009992 thru 0x00d010 (0x367f bytes) 000488 1787602060.949722 mapread 0x0023f3 thru 0x0037bd (0x13cb bytes) 000489 1787602060.957968 mapread 0x009d30 thru 0x012f42 (0x9213 bytes) 000492 1787602060.960152 trunc from 0x012f43 to 0x0040c9 (0xee79 bytes) 000497 1787602060.984046 read 0x003f12 thru 0x0040c9 (0x1b8 bytes) 000500 1787602060.991598 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40ca): Mirror 8: 0x74069999 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40ca): Mirror 8: 0x74069999 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40ca: skipped 000501 1787602061.214917 mapwrite 0x01bf05 thru 0x025ef5 (0x9ff1 bytes) 000503 1787602061.294122 read 0x01a3fa thru 0x01e145 (0x3d4c bytes) 000504 1787602061.295744 mapwrite 0x00abbb thru 0x00e5bf (0x3a05 bytes) 000505 1787602061.347697 mapwrite 0x019b06 thru 0x022cb9 (0x91b4 bytes) 000507 1787602061.382099 mapwrite 0x013d3e thru 0x01ca9c (0x8d5f bytes) 000511 1787602061.428041 mapread 0x024265 thru 0x025cbc (0x1a58 bytes) 000513 1787602061.429016 write 0x01f3af thru 0x029e69 (0xaabb bytes) 000514 1787602061.431368 write 0x00e9a8 thru 0x01979e (0xadf7 bytes) 000516 1787602061.433149 read 0x007b62 thru 0x00bbaa (0x4049 bytes) 000520 1787602061.436869 mapread 0x004152 thru 0x0071e0 (0x308f bytes) 000521 1787602061.438148 trunc from 0x029e6a to 0x008d5b (0x2110e bytes) 000523 1787602061.451522 trunc from 0x008d5c to 0x03dd07 (0x34fac bytes) 000524 1787602061.472171 mapwrite 0x037a90 thru 0x03ffff (0x8570 bytes) 000525 1787602061.545602 read 0x00ec2b thru 0x0118a5 (0x2c7b bytes) 000526 1787602061.546202 mapread 0x02ff1a thru 0x03a768 (0xa84f bytes) 000527 1787602061.548407 trunc from 0x040000 to 0x0328c5 (0xd73a bytes) 000528 1787602061.565645 write 0x009a54 thru 0x012426 (0x89d3 bytes) 000529 1787602061.571538 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x328c6): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x328c6): Mirror 8: 0x69ce8e49 (stale) CRC-32 checksum value for chunk [0, 0x328c6): Mirror 1: 0x38072da9 Mirror 9: 0x38072da9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x328c6: skipped 000531 1787602062.654171 read 0x006afb thru 0x008f49 (0x244f bytes) 000532 1787602062.663301 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x328c6): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x328c6): Mirror 8: 0x69ce8e49 (stale) CRC-32 checksum value for chunk [0, 0x328c6): Mirror 1: 0x38072da9 Mirror 9: 0x38072da9 Mirror 10: 0x38072da9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x328c6: skipped 000533 1787602063.774880 trunc from 0x0328c6 to 0x0349d5 (0x2110 bytes) 000534 1787602063.856075 read 0x01bdc5 thru 0x022759 (0x6995 bytes) 000537 1787602063.864706 read 0x00902c thru 0x009bee (0xbc3 bytes) 000538 1787602063.883310 write 0x0183c4 thru 0x01af07 (0x2b44 bytes) 000540 1787602063.884741 write 0x00e48f thru 0x00f30e (0xe80 bytes) 000541 1787602063.889534 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x349d6): Mirror 9: 0xdb7b7407 (stale) Mirror 10: 0xdb7b7407 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x349d6): Mirror 9: 0xdb7b7407 (stale) Mirror 10: 0xdb7b7407 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x349d6: skipped 000544 1787602064.179493 mapwrite 0x017f6a thru 0x0212d6 (0x936d bytes) 000545 1787602064.244386 trunc from 0x0349d6 to 0x01a3d0 (0x1a605 bytes) 000546 1787602064.267875 MIRROR_RESYNC 3 mirrors 000547 1787602064.984023 MIRROR_EXTEND to 4 mirrors 000549 1787602065.569094 write 0x002902 thru 0x00bb51 (0x9250 bytes) 000551 1787602065.640085 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a3d1): Mirror 10: 0x3554809b (stale) Mirror 11: 0x3554809b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a3d1): Mirror 10: 0x3554809b (stale) Mirror 11: 0x3554809b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a3d1: skipped 000552 1787602065.830265 mapread 0x015868 thru 0x01a3d0 (0x4b69 bytes) 000553 1787602065.845820 write 0x030d9b thru 0x0339a6 (0x2c0c bytes) 000554 1787602065.848680 trunc from 0x0339a7 to 0x02bbe9 (0x7dbd bytes) 000555 1787602065.865596 trunc from 0x02bbea to 0x02418e (0x7a5b bytes) 000556 1787602065.891722 MIRROR_RESYNC 3 mirrors 000558 1787602066.732598 mapwrite 0x03c288 thru 0x03ffff (0x3d78 bytes) 000559 1787602066.949764 write 0x01fc38 thru 0x024aed (0x4eb6 bytes) 000560 1787602066.965277 trunc from 0x040000 to 0x0011a6 (0x3ee59 bytes) 000561 1787602066.988360 mapread 0x0001eb thru 0x0011a6 (0xfbc bytes) 000563 1787602066.995746 mapwrite 0x006ff2 thru 0x00d6c2 (0x66d1 bytes) 000564 1787602067.083015 mapwrite 0x033160 thru 0x039539 (0x63da bytes) 000566 1787602067.218216 write 0x034e00 thru 0x03e326 (0x9527 bytes) 000568 1787602067.224276 mapwrite 0x0035c4 thru 0x011423 (0xde60 bytes) 000569 1787602067.292802 mapread 0x011c6e thru 0x018f08 (0x729b bytes) 000570 1787602067.301714 mapwrite 0x0364cf thru 0x038325 (0x1e57 bytes) 000571 1787602067.340806 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e327): Mirror 10: 0x1ae02e95 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e327): Mirror 10: 0x1ae02e95 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e327: skipped 000573 1787602067.608074 trunc from 0x03e327 to 0x03520b (0x911b bytes) 000577 1787602067.627224 write 0x01e38d thru 0x01fc50 (0x18c4 bytes) 000578 1787602067.635207 mapwrite 0x01b882 thru 0x022305 (0x6a84 bytes) 000582 1787602067.689609 read 0x02b0c3 thru 0x03520b (0xa149 bytes) 000583 1787602067.702037 mapread 0x00caa6 thru 0x00f6af (0x2c0a bytes) 000585 1787602067.715563 trunc from 0x03520c to 0x00a855 (0x2a9b6 bytes) 000587 1787602067.735729 trunc from 0x00a856 to 0x009e77 (0x9de bytes) 000588 1787602067.755268 trunc from 0x009e78 to 0x005021 (0x4e56 bytes) 000589 1787602067.777380 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) 000590 1787602067.966496 MIRROR_EXTEND to 2 mirrors 000591 1787602068.633722 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) 000593 1787602068.774249 read 0x000355 thru 0x005021 (0x4ccd bytes) 000599 1787602068.789157 read 0x0015de thru 0x005021 (0x3a44 bytes) 000600 1787602068.789594 trunc from 0x005022 to 0x02b14c (0x2612b bytes) 000601 1787602068.808064 trunc from 0x02b14d to 0x03e137 (0x12feb bytes) 000602 1787602068.823680 mapwrite 0x00cbbf thru 0x014b08 (0x7f4a bytes) 000603 1787602068.883894 write 0x023b4c thru 0x02df28 (0xa3dd bytes) 000605 1787602068.886599 mapread 0x038f43 thru 0x03e137 (0x51f5 bytes) 000606 1787602068.887728 mapread 0x0042b8 thru 0x008d1e (0x4a67 bytes) 000608 1787602068.889939 trunc from 0x03e138 to 0x010ea0 (0x2d297 bytes) 000611 1787602068.906448 read 0x002464 thru 0x0037dc (0x1379 bytes) 000615 1787602068.908957 mapread 0x0054d6 thru 0x006aff (0x162a bytes) 000617 1787602068.911160 write 0x03b738 thru 0x03e64e (0x2f17 bytes) 000619 1787602068.913006 mapread 0x0110e1 thru 0x01b52f (0xa44f bytes) 000620 1787602068.919968 write 0x0268e4 thru 0x02ab35 (0x4252 bytes) 000621 1787602068.920950 write 0x026d57 thru 0x036008 (0xf2b2 bytes) 000622 1787602068.922225 trunc from 0x03e64f to 0x03df57 (0x6f7 bytes) 000624 1787602068.942973 mapread 0x008a13 thru 0x013a16 (0xb004 bytes) 000626 1787602068.945509 write 0x021ece thru 0x027558 (0x568b bytes) 000630 1787602068.948897 read 0x016c3d thru 0x01c155 (0x5519 bytes) 000631 1787602068.949971 mapread 0x026712 thru 0x0361bb (0xfaaa bytes) 000632 1787602068.950973 read 0x02d594 thru 0x0334d8 (0x5f45 bytes) 000636 1787602068.954331 mapread 0x002a98 thru 0x010ee1 (0xe44a bytes) 000637 1787602068.959138 MIRROR_EXTEND to 2 mirrors 000638 1787602069.588774 write 0x0328fd thru 0x0335c8 (0xccc bytes) 000642 1787602069.654968 mapwrite 0x0389a1 thru 0x03ffff (0x765f bytes) 000643 1787602069.735980 read 0x010121 thru 0x01f3d4 (0xf2b4 bytes) 000645 1787602069.745310 mapwrite 0x039a14 thru 0x03bba2 (0x218f bytes) 000646 1787602069.775261 write 0x00d31e thru 0x0127c6 (0x54a9 bytes) 000648 1787602069.780319 trunc from 0x040000 to 0x01b1d3 (0x24e2c bytes) 000649 1787602069.799724 MIRROR_RESYNC 2 mirrors 000650 1787602070.352492 mapread 0x017c0d thru 0x01b1d3 (0x35c7 bytes) 000653 1787602070.373875 write 0x00dd0f thru 0x00ee63 (0x1155 bytes) 000654 1787602070.444017 trunc from 0x01b1d4 to 0x00ad17 (0x104bc bytes) 000656 1787602070.466175 mapread 0x000ebb thru 0x00ad17 (0x9e5d bytes) 000657 1787602070.475701 trunc from 0x00ad18 to 0x01f821 (0x14b0a bytes) 000658 1787602070.492585 mapwrite 0x017970 thru 0x027424 (0xfab5 bytes) 000659 1787602070.578346 trunc from 0x027425 to 0x0007db (0x26c49 bytes) 000662 1787602070.598594 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x7dc): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7dc): Mirror 2: 0xdbf653e3 (stale) CRC-32 checksum value for chunk [0, 0x7dc): Mirror 1: 0xdbf653e3 Mirror 3: 0xdbf653e3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7dc: skipped 000663 1787602071.198678 trunc from 0x0007dc to 0x00ced3 (0xc6f8 bytes) 000665 1787602071.286057 write 0x03a79f thru 0x03ffff (0x5861 bytes) 000668 1787602071.288191 write 0x0341a5 thru 0x03ffff (0xbe5b bytes) 000669 1787602071.296620 mapwrite 0x02aecb thru 0x02f0b8 (0x41ee bytes) 000675 1787602071.362156 mapread 0x00e77a thru 0x018027 (0x98ae bytes) 000676 1787602071.384011 MIRROR_RESYNC 3 mirrors 000677 1787602072.106739 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) 000678 1787602072.319257 MIRROR_EXTEND to 3 mirrors 000680 1787602073.036509 read 0x03c049 thru 0x03ffff (0x3fb7 bytes) 000681 1787602073.044576 mapread 0x007f56 thru 0x008f9e (0x1049 bytes) 000683 1787602073.054039 mapwrite 0x023b64 thru 0x028db6 (0x5253 bytes) 000688 1787602073.227442 mapread 0x024fa0 thru 0x02f23b (0xa29c bytes) 000692 1787602073.240182 trunc from 0x040000 to 0x014fe6 (0x2b019 bytes) 000694 1787602073.261657 write 0x0135bd thru 0x013c82 (0x6c6 bytes) 000695 1787602073.268259 read 0x003749 thru 0x0085ba (0x4e72 bytes) 000697 1787602073.286600 MIRROR_RESYNC 3 mirrors 000699 1787602074.037795 mapwrite 0x004705 thru 0x00e4ec (0x9de8 bytes) 000700 1787602074.161012 MIRROR_RESYNC 3 mirrors 000701 1787602074.745208 read 0x0014fc thru 0x00dd19 (0xc81e bytes) 000702 1787602074.757913 mapread 0x008aa8 thru 0x00cbe7 (0x4140 bytes) 000708 1787602074.760876 read 0x00fe7a thru 0x014fe6 (0x516d bytes) 000709 1787602074.773460 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) 000712 1787602075.031542 write 0x03bb08 thru 0x03e864 (0x2d5d bytes) 000713 1787602075.119213 mapwrite 0x039137 thru 0x03ffff (0x6ec9 bytes) 000715 1787602075.259478 trunc from 0x040000 to 0x0199eb (0x26614 bytes) 000716 1787602075.284440 mapread 0x00d392 thru 0x0135ca (0x6239 bytes) 000717 1787602075.300550 read 0x012172 thru 0x018917 (0x67a6 bytes) 000718 1787602075.307051 mapread 0x00fb3c thru 0x015f28 (0x63ed bytes) 000721 1787602075.317766 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) 000722 1787602075.474290 trunc from 0x0199ec to 0x033b3e (0x1a153 bytes) 000724 1787602075.489284 read 0x022ca4 thru 0x02374c (0xaa9 bytes) 000725 1787602075.495452 write 0x00d470 thru 0x0166c7 (0x9258 bytes) 000726 1787602075.504272 write 0x03e18f thru 0x03ffff (0x1e71 bytes) 000729 1787602075.507577 mapread 0x0383b4 thru 0x03cffe (0x4c4b bytes) 000733 1787602075.524967 read 0x02e0e4 thru 0x02f935 (0x1852 bytes) 000735 1787602075.545593 write 0x03f6b6 thru 0x03ffff (0x94a bytes) 000736 1787602075.546425 trunc from 0x040000 to 0x0350c2 (0xaf3d bytes) 000737 1787602075.561473 trunc from 0x0350c3 to 0x024c99 (0x10429 bytes) 000739 1787602075.582599 mapread 0x0219ba thru 0x024c99 (0x32e0 bytes) 000740 1787602075.583638 mapwrite 0x01ed79 thru 0x02008a (0x1312 bytes) 000743 1787602075.655437 read 0x014da4 thru 0x01c42e (0x768b bytes) 000746 1787602075.658196 mapread 0x00e1c7 thru 0x0164dd (0x8317 bytes) 000747 1787602075.660112 read 0x01cfba thru 0x01d0ed (0x134 bytes) 000749 1787602075.662108 mapread 0x01bf15 thru 0x024c99 (0x8d85 bytes) 000753 1787602075.669765 MIRROR_EXTEND to 2 mirrors 000754 1787602076.083969 mapread 0x00c2f5 thru 0x01a18e (0xde9a bytes) 000755 1787602076.106124 MIRROR_EXTEND to 3 mirrors 000758 1787602076.573100 read 0x013f88 thru 0x0140c3 (0x13c bytes) 000759 1787602076.578415 mapwrite 0x030beb thru 0x035a72 (0x4e88 bytes) 000760 1787602076.710979 mapwrite 0x000510 thru 0x0100f6 (0xfbe7 bytes) 000761 1787602076.775057 mapwrite 0x0005fa thru 0x008134 (0x7b3b bytes) 000763 1787602076.806966 write 0x00fd6f thru 0x01c304 (0xc596 bytes) 000764 1787602076.811219 trunc from 0x035a73 to 0x026a00 (0xf072 bytes) 000766 1787602076.834802 trunc from 0x026a01 to 0x028438 (0x1a38 bytes) 000767 1787602076.849944 read 0x027b08 thru 0x028438 (0x931 bytes) 000768 1787602076.859445 mapread 0x011181 thru 0x0175ed (0x646d bytes) 000770 1787602076.861550 trunc from 0x028439 to 0x000299 (0x2819f bytes) 000773 1787602076.881702 mapwrite 0x027a6d thru 0x02c82a (0x4dbe bytes) 000774 1787602076.958235 MIRROR_RESYNC 3 mirrors 000775 1787602077.750391 mapread 0x014a5e thru 0x01d9d6 (0x8f79 bytes) 000776 1787602077.768895 write 0x01cdba thru 0x0240b4 (0x72fb bytes) 000777 1787602077.836233 trunc from 0x02c82b to 0x01c0ca (0x10760 bytes) 000778 1787602077.856619 trunc from 0x01c0cb to 0x03de0b (0x21d41 bytes) 000779 1787602077.877885 mapread 0x026ea8 thru 0x02ad28 (0x3e81 bytes) 000784 1787602077.896486 trunc from 0x03de0c to 0x014150 (0x29cbb bytes) 000787 1787602077.920685 mapwrite 0x016391 thru 0x0240da (0xdd4a bytes) 000789 1787602078.003764 mapread 0x0182a5 thru 0x0240da (0xbe36 bytes) 000792 1787602078.005551 read 0x015681 thru 0x01fd10 (0xa690 bytes) 000793 1787602078.006131 read 0x00f59d thru 0x01ed99 (0xf7fd bytes) 000794 1787602078.008537 write 0x006ef0 thru 0x009475 (0x2586 bytes) 000798 1787602078.024486 mapwrite 0x00f503 thru 0x01eea2 (0xf9a0 bytes) 000799 1787602078.071169 read 0x01a9cf thru 0x0240da (0x970c bytes) 000801 1787602078.072743 write 0x02605a thru 0x034058 (0xdfff bytes) 000804 1787602078.075926 read 0x0123ed thru 0x01b043 (0x8c57 bytes) 000805 1787602078.077037 mapread 0x01f601 thru 0x02eb2a (0xf52a bytes) 000806 1787602078.086177 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x34059): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34059): Mirror 2: 0x781994cb (stale) Mirror 3: 0x781994cb (stale) CRC-32 checksum value for chunk [0, 0x34059): Mirror 1: 0x2d4b1e4e Mirror 4: 0x2d4b1e4e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34059: skipped 000807 1787602078.761351 trunc from 0x034059 to 0x01748b (0x1cbcd bytes) 000810 1787602078.895966 mapwrite 0x026f4c thru 0x02dfcf (0x7084 bytes) 000814 1787602078.994205 mapread 0x010ef9 thru 0x017af5 (0x6bfd bytes) 000816 1787602079.019886 MIRROR_RESYNC 4 mirrors 000819 1787602080.110396 MIRROR_RESYNC 4 mirrors 000820 1787602080.433544 trunc from 0x02dfd0 to 0x01b95f (0x12670 bytes) 000821 1787602080.530584 mapwrite 0x027312 thru 0x02cd19 (0x5a08 bytes) 000823 1787602080.606597 read 0x0001a8 thru 0x007916 (0x776f bytes) 000825 1787602080.622187 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cd1a): Mirror 3: 0x41e08431 (stale) Mirror 4: 0x41e08431 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cd1a): Mirror 3: 0x41e08431 (stale) Mirror 4: 0x41e08431 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cd1a: skipped 000828 1787602080.922389 read 0x002c8c thru 0x008da1 (0x6116 bytes) 000830 1787602080.938987 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2cd1a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cd1a): Mirror 3: 0x41e08431 (stale) Mirror 4: 0x41e08431 (stale) CRC-32 checksum value for chunk [0, 0x2cd1a): Mirror 1: 0xf7b1d6bd Mirror 5: 0xf7b1d6bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cd1a: skipped 000833 1787602081.748527 read 0x0292bf thru 0x02cd19 (0x3a5b bytes) 000834 1787602081.761969 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2cd1a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cd1a): Mirror 3: 0x41e08431 (stale) Mirror 4: 0x41e08431 (stale) CRC-32 checksum value for chunk [0, 0x2cd1a): Mirror 1: 0xf7b1d6bd Mirror 5: 0xf7b1d6bd Mirror 6: 0xf7b1d6bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cd1a: skipped 000835 1787602082.621549 write 0x02b805 thru 0x032226 (0x6a22 bytes) 000836 1787602082.682752 trunc from 0x032227 to 0x003c9a (0x2e58c bytes) 000837 1787602082.703671 read 0x00332b thru 0x003c9a (0x970 bytes) 000840 1787602082.711452 mapread 0x000c73 thru 0x003c9a (0x3028 bytes) 000842 1787602082.724600 mapread 0x00240d thru 0x003c9a (0x188e bytes) 000844 1787602082.726104 write 0x02442e thru 0x031e63 (0xda36 bytes) 000847 1787602082.733501 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31e64): Mirror 4: 0x8920d5d3 (stale) Mirror 5: 0xb5359294 (stale) Mirror 6: 0xb5359294 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31e64): Mirror 4: 0x8920d5d3 (stale) Mirror 5: 0xb5359294 (stale) Mirror 6: 0xb5359294 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31e64: skipped 000850 1787602083.044525 trunc from 0x031e64 to 0x004eef (0x2cf74 bytes) 000852 1787602083.066251 read 0x0031e0 thru 0x004512 (0x1333 bytes) 000853 1787602083.072501 trunc from 0x004ef0 to 0x029722 (0x24833 bytes) 000855 1787602083.087833 mapread 0x005e1a thru 0x00b0e1 (0x52c8 bytes) 000857 1787602083.105922 mapread 0x016af4 thru 0x01ca6a (0x5f77 bytes) 000860 1787602083.116521 write 0x0317e6 thru 0x03ffff (0xe81a bytes) 000863 1787602083.121644 read 0x0336e3 thru 0x034a0f (0x132d bytes) 000865 1787602083.123248 read 0x0143e4 thru 0x022c51 (0xe86e bytes) 000870 1787602083.139616 mapwrite 0x022da7 thru 0x0308d4 (0xdb2e bytes) 000871 1787602083.200536 trunc from 0x040000 to 0x0276e8 (0x18917 bytes) 000873 1787602083.218039 write 0x02292d thru 0x031493 (0xeb67 bytes) 000874 1787602083.220509 read 0x019db9 thru 0x01cff9 (0x3241 bytes) 000877 1787602083.222371 write 0x03eaa6 thru 0x03ffff (0x155a bytes) 000878 1787602083.223806 mapwrite 0x01cb02 thru 0x020eee (0x43ed bytes) 000879 1787602083.266771 read 0x0124a1 thru 0x0159f6 (0x3556 bytes) 000880 1787602083.267413 mapread 0x02eff6 thru 0x0328d9 (0x38e4 bytes) 000883 1787602083.289210 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x981e4c85 (stale) Mirror 5: 0xa404a285 (stale) Mirror 6: 0xa404a285 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9965561b Mirror 7: 0x9965561b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000884 1787602084.116439 mapwrite 0x02e190 thru 0x0389fa (0xa86b bytes) 000889 1787602084.284449 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x981e4c85 (stale) Mirror 5: 0xa404a285 (stale) Mirror 6: 0xa404a285 (stale) Mirror 7: 0x9965561b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9ecfa4b0 Mirror 8: 0x9ecfa4b0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000891 1787602085.115738 mapread 0x03dcc7 thru 0x03ffff (0x2339 bytes) 000892 1787602085.136382 mapread 0x0242ea thru 0x02677e (0x2495 bytes) 000893 1787602085.150401 write 0x00680a thru 0x00f892 (0x9089 bytes) 000895 1787602085.235438 mapread 0x0212e9 thru 0x0234a3 (0x21bb bytes) 000897 1787602085.245440 trunc from 0x040000 to 0x02e9f4 (0x1160b bytes) 000902 1787602085.264939 mapwrite 0x038372 thru 0x03ffff (0x7c8e bytes) 000905 1787602085.399938 MIRROR_RESYNC 6 mirrors 000907 1787602087.037295 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) 000908 1787602087.323517 read 0x029ab9 thru 0x030cce (0x7216 bytes) 000909 1787602087.341133 MIRROR_RESYNC 5 mirrors 000911 1787602087.807415 trunc from 0x040000 to 0x00da09 (0x325f6 bytes) 000913 1787602087.938415 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xda0a): Mirror 5: 0x9ce4a2dd (stale) Mirror 6: 0x9ce4a2dd (stale) Mirror 7: 0x9ce4a2dd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xda0a): Mirror 5: 0x9ce4a2dd (stale) Mirror 6: 0x9ce4a2dd (stale) Mirror 7: 0x9ce4a2dd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xda0a: skipped 000915 1787602088.350129 write 0x01092f thru 0x017afb (0x71cd bytes) 000916 1787602088.355422 read 0x002957 thru 0x00573e (0x2de8 bytes) 000919 1787602088.382764 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17afc): Mirror 5: 0xde01afff (stale) Mirror 7: 0xde01afff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17afc): Mirror 5: 0xde01afff (stale) Mirror 7: 0xde01afff (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17afc: skipped 000920 1787602088.729951 mapwrite 0x03c945 thru 0x03ffff (0x36bb bytes) 000922 1787602088.827967 write 0x028b06 thru 0x02f9ea (0x6ee5 bytes) 000926 1787602088.844269 trunc from 0x040000 to 0x02bc64 (0x1439b bytes) 000932 1787602088.869356 read 0x022d97 thru 0x0266ad (0x3917 bytes) 000936 1787602088.880135 read 0x01d281 thru 0x028161 (0xaee1 bytes) 000938 1787602088.909105 write 0x01c77e thru 0x0273bb (0xac3e bytes) 000939 1787602088.910341 trunc from 0x02bc65 to 0x01c640 (0xf624 bytes) 000940 1787602088.931913 trunc from 0x01c641 to 0x03b3ba (0x1ed7a bytes) 000941 1787602088.961165 mapread 0x00aecb thru 0x01378e (0x88c4 bytes) 000942 1787602088.962501 mapread 0x01030f thru 0x015097 (0x4d89 bytes) 000943 1787602088.964634 read 0x028681 thru 0x037ef7 (0xf877 bytes) 000946 1787602088.983231 write 0x03d205 thru 0x03ffff (0x2dfb bytes) 000947 1787602088.984382 write 0x02c0fc thru 0x035127 (0x902c bytes) 000948 1787602088.985458 mapread 0x00c9c0 thru 0x014988 (0x7fc9 bytes) 000949 1787602088.986582 read 0x00e866 thru 0x0138fd (0x5098 bytes) 000951 1787602088.988948 read 0x03157c thru 0x03dc4e (0xc6d3 bytes) 000953 1787602088.998384 mapread 0x03ca43 thru 0x03ffff (0x35bd bytes) 000957 1787602089.002914 mapread 0x02a761 thru 0x03884e (0xe0ee bytes) 000958 1787602089.009411 mapwrite 0x019622 thru 0x028e8b (0xf86a bytes) 000959 1787602089.104544 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xfdcb2f90 (stale) Mirror 7: 0xfdcb2f90 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2467dcee Mirror 8: 0x2467dcee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000964 1787602089.831694 trunc from 0x040000 to 0x00b2ee (0x34d11 bytes) 000967 1787602089.936979 trunc from 0x00b2ef to 0x037592 (0x2c2a4 bytes) 000969 1787602089.962991 mapwrite 0x02b84b thru 0x036d2f (0xb4e5 bytes) 000972 1787602090.054538 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x37593): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37593): Mirror 5: 0x1d8fd234 (stale) Mirror 7: 0x1d8fd234 (stale) Mirror 8: 0xca673a2f (stale) CRC-32 checksum value for chunk [0, 0x37593): Mirror 1: 0x7ae5ed18 Mirror 9: 0x7ae5ed18 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37593: skipped 000973 1787602090.878074 MIRROR_RESYNC 5 mirrors 000975 1787602091.936418 MIRROR_EXTEND to 6 mirrors 000976 1787602092.681923 mapwrite 0x006852 thru 0x00fd15 (0x94c4 bytes) 000978 1787602093.020793 MIRROR_RESYNC 6 mirrors 000979 1787602094.402280 write 0x00a3e5 thru 0x0197b0 (0xf3cc bytes) 000982 1787602094.480629 read 0x008b76 thru 0x016db7 (0xe242 bytes) 000983 1787602094.486319 write 0x00e86a thru 0x014e38 (0x65cf bytes) 000984 1787602094.487460 trunc from 0x037593 to 0x018ee9 (0x1e6a9 bytes) 000986 1787602094.512362 mapread 0x00326c thru 0x004294 (0x1029 bytes) 000987 1787602094.520499 read 0x00b1cb thru 0x018ee9 (0xdd1f bytes) 000988 1787602094.521345 read 0x018e43 thru 0x018ee9 (0xa7 bytes) 000992 1787602094.524465 mapwrite 0x03bc8a thru 0x03ffff (0x4376 bytes) 000993 1787602094.590793 mapread 0x034462 thru 0x035284 (0xe23 bytes) 000996 1787602094.592628 mapwrite 0x00973c thru 0x012141 (0x8a06 bytes) 000997 1787602094.633689 trunc from 0x040000 to 0x0257d4 (0x1a82b bytes) 000998 1787602094.651730 mapread 0x016d88 thru 0x018033 (0x12ac bytes) All operations completed A-OK! PASS 70a (80s) == sanity-flr test 71: check mirror extend parent fid ==== 16:08:20 (1787602100) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 16:08:23 (1787602103) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.176247 s, 11.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142188 s, 7.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fc:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fc:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x319:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 16:08:33 (1787602113) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre2 Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3007264 .. Extending file size to 6713056 .. Extending file size to 8217600 .. Extending file size to 4279840 .. Extending file size to 4975808 .. Extending file size to 7380480 .. Extending file size to 8305216 .. Extending file size to 8095904 .. Extending file size to 8689504 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8677280 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8765984 .. Extending file size to 9023584 .. Waiting 57499 57501 57502 57504 57506 failed 192.168.202.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre2 (opts:) 192.168.202.109@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (77s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 16:09:51 (1787602191) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre2 Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1567840 .. Extending file size to 2803968 .. failed Extending file size to 2670080 .. Extending file size to 8057120 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6679872 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7508576 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8900896 .. Extending file size to 8162304 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8987968 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59085 59086 59087 59090 59092 done 192.168.202.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre2 (opts:) 192.168.202.109@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 16:11:11 (1787602271) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0894432 s, 23.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.275684 s, 7.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x1c02ee2a (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 16:11:30 (1787602290) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.137211 s, 15.3 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 16:11:37 (1787602297) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x217:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x312:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x312:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x313:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x32e:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 16:11:44 (1787602304) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.178933 s, 11.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x314:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x330:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 16:11:52 (1787602312) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.244231 s, 8.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x315:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x316:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x331:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x315:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x316:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x331:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x332:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 16:11:59 (1787602319) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.219353 s, 9.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x317:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x333:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x317:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x333:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x334:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 16:12:31 (1787602351) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg209-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14652 s, 7.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x318:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x336:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x335:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x318:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x336:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x335:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x319:0x0] } PASS 204d (60s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 16:13:32 (1787602412) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg209-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x337:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x338:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138303 s, 15.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x337:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x338:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x31a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x337:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x338:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x31a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (62s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 16:14:34 (1787602474) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg209-server mds-ost sync done. Creating new pool test_204f oleg209-server: Pool lustre.test_204f created Adding targets to pool oleg209-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg209-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.30176 s, 6.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x339:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x31b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x339:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x31b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg209-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg209-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg209-server: Pool lustre.test_204f destroyed PASS 204f (82s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 16:15:56 (1787602556) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 16:16:03 (1787602563) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 16:16:11 (1787602571) oleg209-server: Pool lustre.test_206 created Adding targets to pool oleg209-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg209-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg209-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg209-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg209-server: Pool lustre.test_206 destroyed PASS 206 (24s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 16:16:35 (1787602595) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628565 s, 16.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039302 s, 26.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xee01be5a Mirror 3: 0xee01be5a PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 16:16:46 (1787602606) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 16:16:49 (1787602609) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 16:16:53 (1787602613) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.11487 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x9b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1d40000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1d40000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1d40000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x23f0000 Read this (4 bytes): BBBB 192.168.202.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 16:17:31 (1787602651) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0336735 s, 243 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 16:17:37 (1787602657) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439365 s, 23.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x23c lmm_fid: [0x200000404:0x23c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x23c lmm_fid: [0x200000404:0x23c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x325:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834492 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-24 16:17:39.000000000 -0400 Modify: 2026-08-24 16:17:39.000000000 -0400 Change: 2026-08-24 16:17:40.000000000 -0400 Birth: 2026-08-24 16:17:39.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 16:17:46 (1787602666) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 16:17:49 (1787602669) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.393801 s, 104 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0754662 s, 54.3 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 24 16:17 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23e:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 24 16:17 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 16:17:57 (1787602677) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 16:18:04 (1787602684) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1686 sec =========== 16:18:12 (1787602692) === sanity-flr: start cleanup 16:18:13 (1787602693) === === sanity-flr: finish cleanup 16:18:19 (1787602699) ===