-----============= acceptance-small: sanity-flr ============----- Sat Aug 22 06:20:05 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg231-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 06:20:27 (1787394027) === oleg231-client.virtnet: executing check_config_client /mnt/lustre oleg231-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg231-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff99c5c7c15800.idle_timeout=debug osc.lustre-OST0001-osc-ffff99c5c7c15800.idle_timeout=debug disable quota as required oleg231-server: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 06:21:07 (1787394067) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7624] == sanity-flr test 0a: lfs mirror create with -N option == 06:21:10 (1787394070) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 06:21:26 (1787394086) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 06:21:29 (1787394089) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 06:21:32 (1787394092) PASS 0d (32s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 06:22:04 (1787394124) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 06:22:08 (1787394128) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 06:22:11 (1787394131) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff99c5c7c15800.stats=clear osc.lustre-OST0001-osc-ffff99c5c7c15800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 06:22:20 (1787394140) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 06:22:32 (1787394152) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 06:22:39 (1787394159) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 06:22:43 (1787394163) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 06:22:52 (1787394172) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 06:23:02 (1787394182) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 06:23:09 (1787394189) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 06:23:12 (1787394192) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 06:23:19 (1787394199) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 06:23:31 (1787394211) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 06:23:38 (1787394218) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 3.68009 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236625 s, 4.4 MB/s PASS 21 (17s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 06:23:56 (1787394236) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.502761 s, 31.3 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 06:24:06 (1787394246) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 06:24:18 (1787394258) Stopping /mnt/lustre-ost1 (opts:) on oleg231-server oleg231-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99c5c7c15800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99c5c7c15800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg231-server: oleg231-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg231-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg231-server oleg231-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99c5c7c15800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99c5c7c15800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg231-server: oleg231-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg231-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (84s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 06:25:42 (1787394342) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.320436 s, 29.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg231-server oleg231-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99c5c7c15800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99c5c7c15800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg231-server: oleg231-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg231-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (45s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 06:26:27 (1787394387) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg231-server oleg231-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99c5c7c15800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99c5c7c15800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg231-server: oleg231-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg231-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg231-server oleg231-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99c5c7c15800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99c5c7c15800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg231-server: oleg231-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg231-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (77s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 06:27:44 (1787394464) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.17155 s, 10.1 MB/s -rw-r--r-- 1 root root 52428800 Aug 22 06:27 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.063s user 0m0.001s sys 0m0.178s Stopping /mnt/lustre-ost1 (opts:) on oleg231-server oleg231-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99c5c7c15800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99c5c7c15800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.070s user 0m0.002s sys 0m0.178s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg231-server: oleg231-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg231-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg231-server oleg231-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99c5c7c15800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99c5c7c15800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.456s user 0m0.000s sys 0m0.189s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg231-server: oleg231-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg231-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg231-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99c5c7c15800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99c5c7c15800.ost_server_uuid in FULL state after 0 sec PASS 33b (103s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 06:29:27 (1787394567) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 06:29:31 (1787394571) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 06:29:36 (1787394576) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (5s) == sanity-flr test 35: allow to write to mirrored files == 06:29:40 (1787394580) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 06:29:49 (1787394589) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.754408 s, 16.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 06:30:04 (1787394604) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367016 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523748 s, 20.0 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 06:30:13 (1787394613) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 06:30:23 (1787394623) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0018308 s, 559 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00149612 s, 684 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00143744 s, 712 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00194432 s, 527 kB/s flv=10 olv=10 PASS 36d (13s) == sanity-flr test 37: mirror I/O API verification ======= 06:30:36 (1787394636) create test files with size 15747746 .. efee48990492eaa6166190e83dd3cf41 - 8a2c1b268f7068afc688175b4bba399e - e7975bfebe9b7ea2bd9e7db0a74c6422 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg231-client.virtnet /mnt/lustre (opts:) Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre PASS 37 (28s) == sanity-flr test 38: resync ============================ 06:31:04 (1787394664) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 06:31:18 (1787394678) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 06:31:26 (1787394686) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847359 s, 12.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505452 s, 20.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcb:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcc:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 06:31:35 (1787394695) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.17625 s, 11.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.586425 s, 7.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.451954 s, 4.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (22s) == sanity-flr test 42: lfs mirror verify ================= 06:31:57 (1787394717) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (5s) == sanity-flr test 43a: mirror pick on write ============= 06:32:02 (1787394722) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (5s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 06:32:07 (1787394727) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447187 s, 23.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 06:32:17 (1787394737) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 06:32:20 (1787394740) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd1:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd1:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 06:32:31 (1787394751) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.401289 s, 26.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg231-client.virtnet /mnt/lustre (opts:) Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 06:32:44 (1787394764) llite.lustre-ffff99c5d9234000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.301501 s, 34.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg231-client.virtnet /mnt/lustre (opts:) Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff99c5c34e1800.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 06:33:01 (1787394781) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (41s) == sanity-flr test 46: Verify setstripe --copy option ==== 06:33:43 (1787394823) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (32s) == sanity-flr test 47: Verify mirror obj alloc =========== 06:34:15 (1787394855) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 06:34:18 (1787394858) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.217513 s, 14.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.178658 s, 17.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xaf4f2c38 (nosync,stale) Mirror 2: 0xaf4f2c38 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 62583289e3a985138d9537ee59255fc4 - mirror 1 checksum: 15a516ea26a8429ac2f30494693b8173 - mirror 2 checksum: 62583289e3a985138d9537ee59255fc4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-22T06:34:21-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-22T06:34:21-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe7:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 62583289e3a985138d9537ee59255fc4 - mirror 1 checksum: 15a516ea26a8429ac2f30494693b8173 - mirror 2 checksum: 15a516ea26a8429ac2f30494693b8173 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-22T06:34:21-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-22T06:34:21-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe7:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 06:34:32 (1787394872) Failing mds1 on oleg231-server Stopping /mnt/lustre-mds1 (opts:) on oleg231-server 06:34:39 (1787394879) shut down facet: mds1 facet_host: oleg231-server facet_failover_host: oleg231-server Failover mds1 to oleg231-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg231-server: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-MDT0000 06:35:04 (1787394904) targets are mounted 06:35:04 (1787394904) facet_failover done oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369773 s, 28.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } PASS 50A (51s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 06:35:23 (1787394923) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 06:35:26 (1787394926) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 06:35:31 (1787394931) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0132802 s, 1.2 MB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 06:35:40 (1787394940) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 06:35:45 (1787394945) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (5s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 06:35:50 (1787394950) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (28s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 06:36:18 (1787394978) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 06:36:49 (1787395009) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 06:37:14 (1787395034) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 06:37:41 (1787395061) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 06:37:49 (1787395069) mirror create pid 53788 mirror split pid 53790 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53788 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53790 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 06:38:58 (1787395138) Seed set to 5140 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1787395140.949397 write 0x03cf29 thru 0x03ffff (0x30d7 bytes) 000002 1787395140.967793 read 0x02192b thru 0x026c12 (0x52e8 bytes) truncating to largest ever: 0x9941 000003 1787395140.979617 trunc from 0x040000 to 0x009940 (0x366bf bytes) 000004 1787395141.001336 read 0x004eab thru 0x005f04 (0x105a bytes) 000007 1787395141.025583 write 0x019d9c thru 0x021857 (0x7abc bytes) 000008 1787395141.033969 MIRROR_EXTEND to 2 mirrors 000009 1787395141.926054 mapwrite 0x02cd20 thru 0x039bb4 (0xce95 bytes) 000010 1787395142.208571 mapwrite 0x016a67 thru 0x01d583 (0x6b1d bytes) 000011 1787395142.260498 write 0x0092f8 thru 0x0142dc (0xafe5 bytes) 000012 1787395142.265903 read 0x010d13 thru 0x011e44 (0x1132 bytes) 000013 1787395142.266576 read 0x010a79 thru 0x01734b (0x68d3 bytes) 000014 1787395142.267234 mapwrite 0x00b209 thru 0x0129a3 (0x779b bytes) 000016 1787395142.351004 mapwrite 0x031fca thru 0x035fb2 (0x3fe9 bytes) 000017 1787395142.506094 MIRROR_RESYNC 2 mirrors 000018 1787395143.268111 mapread 0x00c79a thru 0x01be0a (0xf671 bytes) 000019 1787395143.298554 write 0x020ed2 thru 0x02fcd4 (0xee03 bytes) truncating to largest ever: 0x1381f 000020 1787395143.381027 trunc from 0x039bb5 to 0x01381e (0x26396 bytes) 000021 1787395143.425361 mapread 0x00cb3e thru 0x01381e (0x6ce1 bytes) 000023 1787395143.436351 write 0x012836 thru 0x01c2df (0x9aaa bytes) 000025 1787395143.439470 read 0x013b2a thru 0x01c2df (0x87b6 bytes) 000026 1787395143.440993 mapread 0x01998c thru 0x01c2df (0x2954 bytes) 000027 1787395143.442943 mapwrite 0x010661 thru 0x018275 (0x7c15 bytes) 000028 1787395143.543282 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1c2e0): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c2e0): Mirror 2: 0x5dc3c78c (stale) CRC-32 checksum value for chunk [0, 0x1c2e0): Mirror 1: 0x96f58d86 Mirror 3: 0x96f58d86 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c2e0: skipped 000032 1787395144.478597 mapread 0x010aa9 thru 0x01229b (0x17f3 bytes) 000034 1787395144.540116 mapwrite 0x014862 thru 0x01c27d (0x7a1c bytes) truncating to largest ever: 0x1e261 000035 1787395144.757661 trunc from 0x01c2e0 to 0x01e260 (0x1f81 bytes) 000036 1787395144.837594 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e261): Mirror 3: 0xa795bf36 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e261): Mirror 3: 0xa795bf36 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e261: skipped 000041 1787395145.376741 write 0x0378b7 thru 0x039f8d (0x26d7 bytes) 000043 1787395145.385374 mapwrite 0x00b5a0 thru 0x00cb5b (0x15bc bytes) 000044 1787395145.454145 write 0x01aa87 thru 0x025561 (0xaadb bytes) 000046 1787395145.467128 read 0x023926 thru 0x02a703 (0x6dde bytes) truncating to largest ever: 0x2deaf 000047 1787395145.481194 trunc from 0x039f8e to 0x02deae (0xc0df bytes) 000048 1787395145.509258 MIRROR_RESYNC 2 mirrors 000049 1787395146.227074 mapread 0x00353c thru 0x00af8e (0x7a53 bytes) 000051 1787395146.257132 write 0x02798b thru 0x035795 (0xde0b bytes) 000052 1787395146.357979 mapwrite 0x036db9 thru 0x03ad33 (0x3f7b bytes) 000053 1787395146.555347 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ad34): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ad34): Mirror 3: 0x6a47d7f4 (stale) CRC-32 checksum value for chunk [0, 0x3ad34): Mirror 1: 0x4fcc97cb Mirror 4: 0x4fcc97cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ad34: skipped 000054 1787395147.784052 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) 000055 1787395148.263299 mapwrite 0x013583 thru 0x019341 (0x5dbf bytes) 000056 1787395148.699258 mapread 0x01e2e9 thru 0x0280d0 (0x9de8 bytes) 000058 1787395148.756331 trunc from 0x03ad34 to 0x001599 (0x3979a bytes) 000060 1787395148.782561 write 0x024bfe thru 0x0332f4 (0xe6f7 bytes) 000062 1787395148.788948 mapread 0x030efb thru 0x032744 (0x184a bytes) 000063 1787395148.790236 mapread 0x01b685 thru 0x0226b6 (0x7032 bytes) 000066 1787395148.804274 trunc from 0x0332f5 to 0x01b1ff (0x180f5 bytes) 000068 1787395148.830555 write 0x020717 thru 0x02987a (0x9164 bytes) 000071 1787395148.836953 read 0x00beba thru 0x01897c (0xcac3 bytes) 000072 1787395148.850002 mapwrite 0x00b300 thru 0x0168c6 (0xb5c7 bytes) 000076 1787395148.915551 write 0x02b982 thru 0x0330fc (0x777b bytes) truncating to largest ever: 0x3f858 000078 1787395148.919920 trunc from 0x0330fd to 0x03f857 (0xc75b bytes) 000079 1787395148.945851 write 0x016b67 thru 0x02414f (0xd5e9 bytes) 000080 1787395148.947622 write 0x0380cb thru 0x03ffff (0x7f35 bytes) 000082 1787395148.961205 read 0x02dd36 thru 0x02e9af (0xc7a bytes) 000084 1787395148.963158 write 0x00186b thru 0x00eeda (0xd670 bytes) 000085 1787395148.976907 write 0x011502 thru 0x014a3e (0x353d bytes) 000086 1787395148.980337 mapread 0x01696c thru 0x018adb (0x2170 bytes) 000087 1787395148.982255 write 0x00e535 thru 0x0132cc (0x4d98 bytes) 000091 1787395148.989334 read 0x01dbb2 thru 0x025a39 (0x7e88 bytes) 000092 1787395148.991321 write 0x01009f thru 0x01f893 (0xf7f5 bytes) 000093 1787395148.994988 mapwrite 0x03affa thru 0x03ffff (0x5006 bytes) 000094 1787395149.067556 trunc from 0x040000 to 0x003f1c (0x3c0e3 bytes) 000095 1787395149.092175 write 0x02abf6 thru 0x030531 (0x593c bytes) 000096 1787395149.095935 read 0x02963b thru 0x030531 (0x6ef7 bytes) 000097 1787395149.107426 mapread 0x011437 thru 0x0201d2 (0xed9c bytes) 000098 1787395149.110951 read 0x017026 thru 0x023ec6 (0xcea1 bytes) 000099 1787395149.112348 read 0x007475 thru 0x0128a2 (0xb42e bytes) 000100 1787395149.113368 mapwrite 0x01d3ac thru 0x01f78d (0x23e2 bytes) 000101 1787395149.158825 mapwrite 0x03cec4 thru 0x03ffff (0x313c bytes) 000102 1787395149.263008 write 0x005f2f thru 0x0144b3 (0xe585 bytes) 000103 1787395149.267102 read 0x01ac73 thru 0x01cd1f (0x20ad bytes) 000109 1787395149.272302 write 0x0347f5 thru 0x03ffff (0xb80b bytes) 000110 1787395149.274551 write 0x00b43a thru 0x0125c4 (0x718b bytes) 000113 1787395149.295321 read 0x0305d3 thru 0x03926a (0x8c98 bytes) 000115 1787395149.298756 mapwrite 0x025e2c thru 0x02d3a6 (0x757b bytes) 000117 1787395149.392956 mapwrite 0x02a78a thru 0x03316c (0x89e3 bytes) 000118 1787395149.441348 read 0x02eb0c thru 0x03bd0c (0xd201 bytes) 000119 1787395149.442825 write 0x006823 thru 0x0167b7 (0xff95 bytes) 000120 1787395149.446117 trunc from 0x040000 to 0x020e48 (0x1f1b7 bytes) 000121 1787395149.469116 write 0x026325 thru 0x0299c0 (0x369c bytes) 000122 1787395149.477335 MIRROR_RESYNC 2 mirrors 000123 1787395150.066849 mapwrite 0x01fa71 thru 0x026eed (0x747d bytes) 000126 1787395150.417469 mapwrite 0x01869a thru 0x021e07 (0x976e bytes) 000128 1787395150.638732 read 0x01d601 thru 0x02108c (0x3a8c bytes) 000129 1787395150.649155 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x299c1): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x299c1): Mirror 4: 0xb28af04d (stale) CRC-32 checksum value for chunk [0, 0x299c1): Mirror 1: 0xeef92213 Mirror 5: 0xeef92213 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x299c1: skipped 000131 1787395152.498460 mapread 0x018cec thru 0x01f107 (0x641c bytes) 000133 1787395152.591838 trunc from 0x0299c1 to 0x0071d7 (0x227e9 bytes) 000134 1787395152.750122 mapwrite 0x000c3e thru 0x00e7b1 (0xdb74 bytes) 000135 1787395152.912969 trunc from 0x00e7b2 to 0x009446 (0x536b bytes) 000139 1787395152.953072 mapread 0x002a63 thru 0x00865e (0x5bfc bytes) 000140 1787395152.955530 read 0x008128 thru 0x009446 (0x131f bytes) 000141 1787395152.960152 mapread 0x000e76 thru 0x009446 (0x85d1 bytes) 000143 1787395152.968404 write 0x00d285 thru 0x00ea8d (0x1809 bytes) 000144 1787395152.997719 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xea8e): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xea8e): Mirror 4: 0x67eb70b8 (stale) Mirror 5: 0x67eb70b8 (stale) CRC-32 checksum value for chunk [0, 0xea8e): Mirror 1: 0xdd9cdabc Mirror 6: 0xdd9cdabc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xea8e: skipped 000145 1787395154.488230 read 0x00e455 thru 0x00ea8d (0x639 bytes) 000146 1787395154.503407 mapwrite 0x0237b7 thru 0x0239b7 (0x201 bytes) 000148 1787395154.718196 mapread 0x001aca thru 0x01040e (0xe945 bytes) 000149 1787395154.761867 mapread 0x00e775 thru 0x00fe1c (0x16a8 bytes) 000150 1787395154.763306 mapwrite 0x0384b8 thru 0x03ffff (0x7b48 bytes) 000151 1787395154.998862 write 0x03891e thru 0x03ffff (0x76e2 bytes) 000152 1787395155.000364 read 0x00d062 thru 0x0183f1 (0xb390 bytes) 000153 1787395155.030847 mapread 0x03ff3f thru 0x03ffff (0xc1 bytes) 000154 1787395155.034368 mapread 0x014d8f thru 0x015bbb (0xe2d bytes) 000155 1787395155.040255 read 0x01c3be thru 0x02885a (0xc49d bytes) 000156 1787395155.055837 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x242c1705 (stale) Mirror 6: 0xe7d65bf1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x242c1705 (stale) Mirror 6: 0xe7d65bf1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000159 1787395155.445774 write 0x023f7a thru 0x0266ba (0x2741 bytes) 000160 1787395155.479394 write 0x0071c0 thru 0x00b51f (0x4360 bytes) 000161 1787395155.494443 read 0x011555 thru 0x018fc5 (0x7a71 bytes) 000163 1787395155.517848 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x242c1705 (stale) Mirror 6: 0xe7d65bf1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1d7489ea Mirror 7: 0x1d7489ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000164 1787395156.645480 mapread 0x005ba9 thru 0x013aa4 (0xdefc bytes) 000165 1787395156.740187 MIRROR_RESYNC 4 mirrors 000167 1787395157.959841 MIRROR_RESYNC 4 mirrors 000169 1787395158.543047 mapread 0x02c160 thru 0x0390be (0xcf5f bytes) 000170 1787395158.582972 MIRROR_EXTEND to 5 mirrors 000174 1787395159.375985 mapread 0x01eced thru 0x02451c (0x5830 bytes) 000175 1787395159.397772 mapread 0x01ba20 thru 0x022d62 (0x7343 bytes) 000176 1787395159.398899 write 0x00159c thru 0x009d6b (0x87d0 bytes) 000179 1787395159.478748 MIRROR_RESYNC 5 mirrors 000180 1787395161.183643 MIRROR_RESYNC 5 mirrors 000181 1787395162.035781 write 0x014a12 thru 0x022e43 (0xe432 bytes) 000183 1787395162.172195 MIRROR_RESYNC 5 mirrors 000186 1787395163.293088 mapread 0x011269 thru 0x01abd2 (0x996a bytes) 000187 1787395163.317566 read 0x029ae3 thru 0x02c93f (0x2e5d bytes) 000191 1787395163.329390 write 0x0001ae thru 0x008c7b (0x8ace bytes) 000192 1787395163.428802 trunc from 0x040000 to 0x01adb9 (0x25246 bytes) 000195 1787395163.484085 mapwrite 0x03bd67 thru 0x03ffff (0x4299 bytes) 000196 1787395163.598111 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x86d3be6c (stale) Mirror 6: 0x86d3be6c (stale) Mirror 8: 0x86d3be6c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x86d3be6c (stale) Mirror 6: 0x86d3be6c (stale) Mirror 8: 0x86d3be6c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000198 1787395164.150098 mapwrite 0x03c65b thru 0x03ffff (0x39a5 bytes) 000199 1787395164.219988 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x86d3be6c (stale) Mirror 6: 0x86d3be6c (stale) Mirror 8: 0x86d3be6c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8529817d Mirror 9: 0x8529817d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000201 1787395164.876073 trunc from 0x040000 to 0x0238a8 (0x1c757 bytes) 000202 1787395164.990074 read 0x0221f1 thru 0x0238a8 (0x16b8 bytes) 000203 1787395164.997174 write 0x01f24b thru 0x026c58 (0x7a0e bytes) 000204 1787395165.003376 mapwrite 0x027ca9 thru 0x02ac1d (0x2f75 bytes) 000205 1787395165.109785 mapwrite 0x01ebb7 thru 0x02aadc (0xbf26 bytes) 000206 1787395165.174714 mapwrite 0x0183d1 thru 0x01876c (0x39c bytes) 000207 1787395165.225849 mapwrite 0x02df3e thru 0x0350d4 (0x7197 bytes) 000208 1787395165.329457 read 0x012636 thru 0x0161d7 (0x3ba2 bytes) 000209 1787395165.341066 mapread 0x003ede thru 0x005664 (0x1787 bytes) 000212 1787395165.347034 read 0x026975 thru 0x0336db (0xcd67 bytes) 000213 1787395165.352746 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x350d5): Mirror 5: 0x5d2407df (stale) Mirror 6: 0x5d2407df (stale) Mirror 8: 0x5d2407df (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x350d5): Mirror 5: 0x5d2407df (stale) Mirror 6: 0x5d2407df (stale) Mirror 8: 0x5d2407df (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x350d5: skipped 000215 1787395165.849919 trunc from 0x0350d5 to 0x00659c (0x2eb38 bytes) 000216 1787395165.875814 read 0x002448 thru 0x00659c (0x4155 bytes) 000217 1787395165.890526 trunc from 0x00659d to 0x00de0f (0x7873 bytes) 000218 1787395165.913373 write 0x007267 thru 0x0156ee (0xe488 bytes) 000221 1787395165.933009 MIRROR_RESYNC 4 mirrors 000222 1787395166.812923 read 0x00fa8c thru 0x0144d8 (0x4a4d bytes) 000224 1787395166.831130 mapread 0x003385 thru 0x00b9ff (0x867b bytes) 000225 1787395166.864874 mapwrite 0x015410 thru 0x0225cf (0xd1c0 bytes) 000226 1787395167.124888 mapread 0x00d5ec thru 0x00f2f7 (0x1d0c bytes) 000227 1787395167.171393 write 0x02bdb5 thru 0x034cd3 (0x8f1f bytes) 000234 1787395167.192187 trunc from 0x034cd4 to 0x03bcc7 (0x6ff4 bytes) 000237 1787395167.231094 trunc from 0x03bcc8 to 0x00052a (0x3b79d bytes) 000238 1787395167.249793 read 0x00034e thru 0x00052a (0x1dd bytes) 000239 1787395167.261052 mapwrite 0x032d82 thru 0x03e3d3 (0xb652 bytes) 000240 1787395167.435558 write 0x027ad1 thru 0x035763 (0xdc93 bytes) 000244 1787395167.457090 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3e3d4): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e3d4): Mirror 5: 0x75411c90 (stale) Mirror 6: 0x75411c90 (stale) Mirror 8: 0x75411c90 (stale) CRC-32 checksum value for chunk [0, 0x3e3d4): Mirror 1: 0xbb28890e Mirror 9: 0xbb28890e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e3d4: skipped 000245 1787395168.107500 write 0x009da5 thru 0x00b784 (0x19e0 bytes) 000247 1787395168.197529 mapread 0x015b01 thru 0x01773c (0x1c3c bytes) 000248 1787395168.259001 mapread 0x026657 thru 0x032a39 (0xc3e3 bytes) 000249 1787395168.306378 read 0x0102cd thru 0x01a7a2 (0xa4d6 bytes) 000250 1787395168.314323 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3e3d4): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e3d4): Mirror 5: 0x75411c90 (stale) Mirror 6: 0x75411c90 (stale) Mirror 8: 0x75411c90 (stale) Mirror 9: 0xbb28890e (stale) CRC-32 checksum value for chunk [0, 0x3e3d4): Mirror 1: 0x7b415a91 Mirror 10: 0x7b415a91 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e3d4: skipped 000252 1787395169.386264 trunc from 0x03e3d4 to 0x003d4e (0x3a685 bytes) 000256 1787395169.526895 read 0x000064 thru 0x003d4e (0x3ceb bytes) 000257 1787395169.536734 read 0x0014cb thru 0x003d4e (0x2884 bytes) 000259 1787395169.538859 write 0x013603 thru 0x01ecf7 (0xb6f5 bytes) 000260 1787395169.541565 mapwrite 0x018476 thru 0x018ab4 (0x63f bytes) 000261 1787395169.608070 read 0x013a7c thru 0x01d226 (0x97ab bytes) 000263 1787395169.610046 mapread 0x004ed3 thru 0x005846 (0x974 bytes) 000267 1787395169.624079 trunc from 0x01ecf8 to 0x008d91 (0x15f66 bytes) 000268 1787395169.646036 read 0x0031f3 thru 0x004439 (0x1247 bytes) 000271 1787395169.648047 write 0x03b6e2 thru 0x03ffff (0x491e bytes) 000272 1787395169.650777 mapread 0x02d202 thru 0x03232f (0x512e bytes) 000273 1787395169.659961 write 0x03ba36 thru 0x03ffff (0x45ca bytes) 000274 1787395169.661458 read 0x022e3c thru 0x024b75 (0x1d3a bytes) 000275 1787395169.662154 trunc from 0x040000 to 0x025c4f (0x1a3b0 bytes) 000276 1787395169.690758 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25c50): Mirror 6: 0xd0ac4dfd (stale) Mirror 8: 0xd0ac4dfd (stale) Mirror 9: 0xe61818fd (stale) Mirror 10: 0xef224dcc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25c50): Mirror 6: 0xd0ac4dfd (stale) Mirror 8: 0xd0ac4dfd (stale) Mirror 9: 0xe61818fd (stale) Mirror 10: 0xef224dcc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25c50: skipped 000278 1787395170.138965 mapwrite 0x02ae39 thru 0x03557e (0xa746 bytes) 000279 1787395170.330874 mapread 0x01042e thru 0x020004 (0xfbd7 bytes) 000281 1787395170.369477 trunc from 0x03557f to 0x01a4f4 (0x1b08a bytes) 000284 1787395170.428975 write 0x024e4d thru 0x02d3dd (0x8591 bytes) 000288 1787395170.442412 read 0x0013e7 thru 0x00bca0 (0xa8ba bytes) 000290 1787395170.467405 trunc from 0x02d3de to 0x0355ae (0x81d1 bytes) 000293 1787395170.511272 read 0x00a9a4 thru 0x0123f8 (0x7a55 bytes) 000294 1787395170.512311 trunc from 0x0355af to 0x02d600 (0x7fae bytes) 000298 1787395170.563355 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d601): Mirror 6: 0x8b6cba31 (stale) Mirror 9: 0x747b659d (stale) Mirror 10: 0xfbd4ea0f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d601): Mirror 6: 0x8b6cba31 (stale) Mirror 9: 0x747b659d (stale) Mirror 10: 0xfbd4ea0f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d601: skipped 000302 1787395171.102953 mapread 0x025253 thru 0x02958e (0x433c bytes) 000304 1787395171.139001 write 0x01736b thru 0x0216a0 (0xa336 bytes) 000305 1787395171.145512 mapwrite 0x0281c6 thru 0x032e5c (0xac97 bytes) 000306 1787395171.276942 MIRROR_RESYNC 4 mirrors 000307 1787395173.046108 MIRROR_EXTEND to 5 mirrors 000311 1787395173.863055 trunc from 0x032e5d to 0x015018 (0x1de44 bytes) 000312 1787395173.953163 mapwrite 0x009fc0 thru 0x00eaeb (0x4b2c bytes) 000314 1787395174.049040 write 0x032745 thru 0x0377c6 (0x5082 bytes) 000315 1787395174.050078 read 0x01a6c1 thru 0x023045 (0x8985 bytes) 000316 1787395174.061751 mapwrite 0x008088 thru 0x00c3c8 (0x4341 bytes) 000317 1787395174.108974 mapread 0x004ec0 thru 0x01346e (0xe5af bytes) 000318 1787395174.110035 mapwrite 0x026a12 thru 0x02bd9a (0x5389 bytes) 000319 1787395174.199013 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x377c7): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x377c7): Mirror 6: 0x2418c3ca (stale) Mirror 9: 0x2418c3ca (stale) Mirror 10: 0x2418c3ca (stale) Mirror 11: 0x2418c3ca (stale) CRC-32 checksum value for chunk [0, 0x377c7): Mirror 1: 0x12e92dc2 Mirror 12: 0x12e92dc2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x377c7: skipped 000321 1787395175.993702 mapwrite 0x035664 thru 0x03c543 (0x6ee0 bytes) 000322 1787395176.462549 mapread 0x0055d2 thru 0x0141d8 (0xec07 bytes) 000323 1787395176.525234 write 0x0005fe thru 0x008c8d (0x8690 bytes) 000327 1787395176.543615 trunc from 0x03c544 to 0x00fc70 (0x2c8d3 bytes) 000328 1787395176.575073 write 0x006e04 thru 0x007868 (0xa65 bytes) 000329 1787395176.576135 mapread 0x00f4f0 thru 0x00fc70 (0x781 bytes) 000330 1787395176.577098 trunc from 0x00fc71 to 0x027e91 (0x18221 bytes) 000331 1787395176.617023 mapread 0x0083f5 thru 0x00f06d (0x6c79 bytes) 000332 1787395176.618500 write 0x0093e1 thru 0x013489 (0xa0a9 bytes) 000334 1787395176.631857 mapread 0x02006f thru 0x0236fa (0x368c bytes) 000336 1787395176.645588 mapread 0x00d784 thru 0x0152dd (0x7b5a bytes) 000340 1787395176.660136 mapread 0x00fff3 thru 0x01844f (0x845d bytes) 000341 1787395176.684828 MIRROR_RESYNC 6 mirrors 000342 1787395178.744813 trunc from 0x027e92 to 0x03d7bf (0x1592e bytes) 000344 1787395178.959795 mapread 0x0192b7 thru 0x028b34 (0xf87e bytes) 000345 1787395179.023421 mapread 0x02b61a thru 0x032a7e (0x7465 bytes) 000346 1787395179.043930 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3d7c0): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d7c0): Mirror 6: 0xab4d0665 (stale) Mirror 9: 0xab4d0665 (stale) Mirror 10: 0xab4d0665 (stale) Mirror 11: 0xab4d0665 (stale) Mirror 12: 0xab4d0665 (stale) CRC-32 checksum value for chunk [0, 0x3d7c0): Mirror 1: 0xab4d0665 Mirror 13: 0xab4d0665 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d7c0: skipped 000347 1787395180.576543 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3d7c0): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d7c0): Mirror 6: 0xab4d0665 (stale) Mirror 9: 0xab4d0665 (stale) Mirror 10: 0xab4d0665 (stale) Mirror 11: 0xab4d0665 (stale) Mirror 12: 0xab4d0665 (stale) CRC-32 checksum value for chunk [0, 0x3d7c0): Mirror 1: 0xab4d0665 Mirror 13: 0xab4d0665 Mirror 14: 0xab4d0665 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d7c0: skipped 000351 1787395182.164507 mapwrite 0x034d6f thru 0x0380f3 (0x3385 bytes) 000352 1787395182.479607 trunc from 0x03d7c0 to 0x01e885 (0x1ef3a bytes) 000353 1787395182.525156 write 0x03d443 thru 0x03ffff (0x2bbd bytes) 000354 1787395182.543052 write 0x004fc8 thru 0x010d2c (0xbd65 bytes) 000356 1787395182.569294 write 0x000867 thru 0x00911f (0x88b9 bytes) 000357 1787395182.601743 trunc from 0x040000 to 0x031f3b (0xe0c4 bytes) 000360 1787395182.663329 mapwrite 0x0107bb thru 0x0203eb (0xfc31 bytes) 000363 1787395182.817186 MIRROR_RESYNC 8 mirrors 000366 1787395184.905036 mapwrite 0x00b600 thru 0x019fe4 (0xe9e5 bytes) 000368 1787395185.479936 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31f3c): Mirror 6: 0xc2cea49e (stale) Mirror 9: 0xc2cea49e (stale) Mirror 10: 0xc2cea49e (stale) Mirror 12: 0xc2cea49e (stale) Mirror 13: 0xc2cea49e (stale) Mirror 14: 0xc2cea49e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31f3c): Mirror 6: 0xc2cea49e (stale) Mirror 9: 0xc2cea49e (stale) Mirror 10: 0xc2cea49e (stale) Mirror 12: 0xc2cea49e (stale) Mirror 13: 0xc2cea49e (stale) Mirror 14: 0xc2cea49e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31f3c: skipped 000370 1787395186.314160 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x31f3c): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31f3c): Mirror 6: 0xc2cea49e (stale) Mirror 9: 0xc2cea49e (stale) Mirror 10: 0xc2cea49e (stale) Mirror 12: 0xc2cea49e (stale) Mirror 13: 0xc2cea49e (stale) Mirror 14: 0xc2cea49e (stale) CRC-32 checksum value for chunk [0, 0x31f3c): Mirror 1: 0x57f5fec0 Mirror 15: 0x57f5fec0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31f3c: skipped 000371 1787395187.577968 trunc from 0x031f3c to 0x0275ae (0xa98d bytes) 000372 1787395187.645598 read 0x00c529 thru 0x00c958 (0x430 bytes) 000373 1787395187.650527 trunc from 0x0275af to 0x0399d8 (0x1242a bytes) 000375 1787395187.678469 MIRROR_RESYNC 8 mirrors 000377 1787395189.488417 write 0x02ce8e thru 0x02f3bd (0x2530 bytes) 000379 1787395189.623733 mapread 0x0264e2 thru 0x026f05 (0xa24 bytes) 000380 1787395189.640286 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x399d9): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x399d9): Mirror 6: 0x6f146c11 (stale) Mirror 9: 0x6f146c11 (stale) Mirror 10: 0x6f146c11 (stale) Mirror 12: 0x6f146c11 (stale) Mirror 13: 0x6f146c11 (stale) Mirror 14: 0x6f146c11 (stale) Mirror 15: 0x6f146c11 (stale) CRC-32 checksum value for chunk [0, 0x399d9): Mirror 1: 0x3528a2b1 Mirror 16: 0x3528a2b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x399d9: skipped 000382 1787395190.814550 read 0x01f27e thru 0x021f98 (0x2d1b bytes) 000384 1787395190.831287 mapread 0x00f78f thru 0x01c2bb (0xcb2d bytes) 000386 1787395190.880982 write 0x01dac7 thru 0x01f84f (0x1d89 bytes) 000388 1787395190.987094 trunc from 0x0399d9 to 0x0211ce (0x1880a bytes) 000390 1787395191.023820 trunc from 0x0211cf to 0x01ba7a (0x5754 bytes) 000392 1787395191.036528 trunc from 0x01ba7b to 0x023d9f (0x8325 bytes) 000394 1787395191.058521 write 0x038c76 thru 0x03ffff (0x738a bytes) 000395 1787395191.059821 mapwrite 0x00e3f3 thru 0x014948 (0x6556 bytes) 000396 1787395191.146567 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xe348a462 (stale) Mirror 10: 0xe348a462 (stale) Mirror 12: 0xe348a462 (stale) Mirror 13: 0xe348a462 (stale) Mirror 14: 0xe348a462 (stale) Mirror 15: 0xe348a462 (stale) Mirror 16: 0x812f95dd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xe348a462 (stale) Mirror 10: 0xe348a462 (stale) Mirror 12: 0xe348a462 (stale) Mirror 13: 0xe348a462 (stale) Mirror 14: 0xe348a462 (stale) Mirror 15: 0xe348a462 (stale) Mirror 16: 0x812f95dd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000397 1787395191.695810 mapread 0x027647 thru 0x02abc3 (0x357d bytes) 000400 1787395191.755880 read 0x03953b thru 0x03ffff (0x6ac5 bytes) 000401 1787395191.772531 mapread 0x01608c thru 0x017b05 (0x1a7a bytes) 000402 1787395191.784906 mapwrite 0x01b59b thru 0x029cdf (0xe745 bytes) 000404 1787395191.904560 trunc from 0x040000 to 0x038ab8 (0x7547 bytes) 000406 1787395191.934335 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38ab9): Mirror 9: 0x2bb892c1 (stale) Mirror 10: 0x2bb892c1 (stale) Mirror 12: 0x2bb892c1 (stale) Mirror 13: 0x2bb892c1 (stale) Mirror 15: 0x2bb892c1 (stale) Mirror 16: 0xb4ac3142 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38ab9): Mirror 9: 0x2bb892c1 (stale) Mirror 10: 0x2bb892c1 (stale) Mirror 12: 0x2bb892c1 (stale) Mirror 13: 0x2bb892c1 (stale) Mirror 15: 0x2bb892c1 (stale) Mirror 16: 0xb4ac3142 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38ab9: skipped 000407 1787395192.487919 mapread 0x000978 thru 0x005e61 (0x54ea bytes) 000410 1787395192.537440 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38ab9): Mirror 9: 0x2bb892c1 (stale) Mirror 10: 0x2bb892c1 (stale) Mirror 12: 0x2bb892c1 (stale) Mirror 15: 0x2bb892c1 (stale) Mirror 16: 0xb4ac3142 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38ab9): Mirror 9: 0x2bb892c1 (stale) Mirror 10: 0x2bb892c1 (stale) Mirror 12: 0x2bb892c1 (stale) Mirror 15: 0x2bb892c1 (stale) Mirror 16: 0xb4ac3142 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38ab9: skipped 000413 1787395192.904045 read 0x006b15 thru 0x0084be (0x19aa bytes) 000415 1787395192.912883 MIRROR_RESYNC 6 mirrors 000416 1787395193.832565 trunc from 0x038ab9 to 0x006060 (0x32a58 bytes) 000417 1787395193.932593 read 0x00602a thru 0x006060 (0x37 bytes) 000418 1787395193.938746 mapread 0x001561 thru 0x006060 (0x4b00 bytes) 000419 1787395193.957360 write 0x01e613 thru 0x02a177 (0xbb65 bytes) 000421 1787395193.965011 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a178): Mirror 10: 0xc181b824 (stale) Mirror 12: 0xc181b824 (stale) Mirror 15: 0xc181b824 (stale) Mirror 16: 0xc181b824 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a178): Mirror 10: 0xc181b824 (stale) Mirror 12: 0xc181b824 (stale) Mirror 15: 0xc181b824 (stale) Mirror 16: 0xc181b824 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a178: skipped 000422 1787395194.746125 trunc from 0x02a178 to 0x002451 (0x27d26 bytes) 000426 1787395194.774550 read 0x0002dc thru 0x002451 (0x2176 bytes) 000429 1787395194.792902 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2452): Mirror 10: 0xdc3495a1 (stale) Mirror 15: 0xdc3495a1 (stale) Mirror 16: 0xdc3495a1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2452): Mirror 10: 0xdc3495a1 (stale) Mirror 15: 0xdc3495a1 (stale) Mirror 16: 0xdc3495a1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2452: skipped 000431 1787395195.191898 mapwrite 0x03d42f thru 0x03ffff (0x2bd1 bytes) 000433 1787395195.300823 read 0x02c0f9 thru 0x02e858 (0x2760 bytes) 000438 1787395195.321318 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x49c0c311 (stale) Mirror 15: 0x49c0c311 (stale) Mirror 16: 0x49c0c311 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6522ccac Mirror 17: 0x6522ccac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000439 1787395196.224797 trunc from 0x040000 to 0x037409 (0x8bf6 bytes) 000442 1787395196.353094 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3740a): Mirror 10: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3740a): Mirror 10: 0xec9e0216 (stale) Mirror 15: 0xec9e0216 (stale) Mirror 16: 0xec9e0216 (stale) Mirror 17: 0xcce8a39e (stale) CRC-32 checksum value for chunk [0, 0x3740a): Mirror 1: 0xcce8a39e Mirror 18: 0xcce8a39e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3740a: skipped 000443 1787395197.761814 write 0x021dd7 thru 0x024ee7 (0x3111 bytes) 000444 1787395197.910088 mapwrite 0x029a8a thru 0x033ab7 (0xa02e bytes) 000446 1787395197.990645 write 0x03e156 thru 0x03ffff (0x1eaa bytes) 000448 1787395197.994837 mapread 0x019459 thru 0x0272e0 (0xde88 bytes) 000449 1787395198.007289 read 0x028ac5 thru 0x03008f (0x75cb bytes) 000451 1787395198.010000 write 0x01c282 thru 0x02858a (0xc309 bytes) 000452 1787395198.017465 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x49c0c311 (stale) Mirror 15: 0x49c0c311 (stale) Mirror 16: 0x49c0c311 (stale) Mirror 17: 0x6522ccac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x49c0c311 (stale) Mirror 15: 0x49c0c311 (stale) Mirror 16: 0x49c0c311 (stale) Mirror 17: 0x6522ccac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000454 1787395198.701504 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x49c0c311 (stale) Mirror 16: 0x49c0c311 (stale) Mirror 17: 0x6522ccac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x49c0c311 (stale) Mirror 16: 0x49c0c311 (stale) Mirror 17: 0x6522ccac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000456 1787395199.378975 mapread 0x006d91 thru 0x00d7be (0x6a2e bytes) 000457 1787395199.425833 mapread 0x02e1ee thru 0x036521 (0x8334 bytes) 000458 1787395199.449437 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x49c0c311 (stale) Mirror 16: 0x49c0c311 (stale) Mirror 17: 0x6522ccac (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4b4c089 Mirror 18: 0x4b4c089 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000459 1787395200.632312 write 0x02dcfd thru 0x030ed8 (0x31dc bytes) 000460 1787395200.748597 trunc from 0x040000 to 0x008e41 (0x371be bytes) 000461 1787395200.773169 read 0x008983 thru 0x008e41 (0x4bf bytes) 000462 1787395200.779260 mapread 0x005803 thru 0x008e41 (0x363f bytes) 000464 1787395200.803495 read 0x005d54 thru 0x005dc2 (0x6f bytes) 000466 1787395200.805406 mapread 0x001992 thru 0x008e41 (0x74b0 bytes) 000467 1787395200.807593 mapread 0x001ef6 thru 0x008e41 (0x6f4c bytes) 000468 1787395200.809328 mapread 0x001f46 thru 0x0036c1 (0x177c bytes) 000469 1787395200.810370 mapread 0x004e3f thru 0x008e41 (0x4003 bytes) 000470 1787395200.811342 mapread 0x005ac6 thru 0x006a93 (0xfce bytes) 000471 1787395200.812197 write 0x0281b4 thru 0x031447 (0x9294 bytes) 000472 1787395200.813920 write 0x037469 thru 0x03ffff (0x8b97 bytes) 000474 1787395200.816395 write 0x025421 thru 0x02f4c0 (0xa0a0 bytes) 000475 1787395200.824966 mapread 0x015947 thru 0x01695c (0x1016 bytes) 000478 1787395200.828909 mapread 0x012a52 thru 0x017450 (0x49ff bytes) 000482 1787395200.834858 mapread 0x03c5c3 thru 0x03ffff (0x3a3d bytes) 000484 1787395200.842461 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x49c0c311 (stale) Mirror 17: 0x6522ccac (stale) Mirror 18: 0x4b4c089 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x49c0c311 (stale) Mirror 17: 0x6522ccac (stale) Mirror 18: 0x4b4c089 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000486 1787395201.267746 read 0x0308a8 thru 0x03fe6f (0xf5c8 bytes) 000490 1787395201.280399 mapwrite 0x03c5eb thru 0x03ffff (0x3a15 bytes) 000491 1787395201.325845 read 0x00f6e5 thru 0x011706 (0x2022 bytes) 000492 1787395201.342974 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x49c0c311 (stale) Mirror 17: 0x6522ccac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x49c0c311 (stale) Mirror 17: 0x6522ccac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000494 1787395201.605855 mapread 0x0283ec thru 0x02a844 (0x2459 bytes) 000495 1787395201.619971 read 0x021a1b thru 0x02f1e0 (0xd7c6 bytes) 000496 1787395201.626975 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x49c0c311 (stale) Mirror 17: 0x6522ccac (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa73cc1b2 Mirror 18: 0xa73cc1b2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000498 1787395202.775848 write 0x032981 thru 0x036bbc (0x423c bytes) 000499 1787395202.996904 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0x6522ccac (stale) Mirror 18: 0xa73cc1b2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0x6522ccac (stale) Mirror 18: 0xa73cc1b2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000500 1787395203.469491 write 0x02efcb thru 0x038e56 (0x9e8c bytes) 000501 1787395203.482393 mapwrite 0x0164dd thru 0x024199 (0xdcbd bytes) 000502 1787395203.637572 mapwrite 0x01642b thru 0x01959d (0x3173 bytes) 000504 1787395203.692322 trunc from 0x040000 to 0x037d64 (0x829b bytes) 000506 1787395203.719505 read 0x02e1eb thru 0x037d64 (0x9b7a bytes) 000507 1787395203.720908 trunc from 0x037d65 to 0x030257 (0x7b0d bytes) 000509 1787395203.749326 write 0x038af4 thru 0x03ffff (0x750c bytes) 000510 1787395203.754791 mapread 0x0087f7 thru 0x010b6f (0x8379 bytes) 000511 1787395203.784326 write 0x026d1a thru 0x02b57b (0x4862 bytes) 000512 1787395203.809788 write 0x03dc3a thru 0x03ffff (0x23c6 bytes) 000513 1787395203.817742 write 0x013c1a thru 0x019888 (0x5c6f bytes) 000514 1787395203.845317 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0xa73cc1b2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0xa73cc1b2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000515 1787395204.454887 write 0x028413 thru 0x02af9a (0x2b88 bytes) 000517 1787395204.485568 read 0x021d8d thru 0x02690c (0x4b80 bytes) 000518 1787395204.505659 MIRROR_RESYNC 2 mirrors 000519 1787395205.756285 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) 000520 1787395205.976306 mapwrite 0x0158c9 thru 0x016c2d (0x1365 bytes) 000522 1787395206.054424 trunc from 0x040000 to 0x0202b0 (0x1fd4f bytes) 000524 1787395206.076367 trunc from 0x0202b1 to 0x0375e0 (0x17330 bytes) 000526 1787395206.094024 write 0x0302b6 thru 0x037982 (0x76cd bytes) 000527 1787395206.105617 read 0x02395f thru 0x025eed (0x258f bytes) 000528 1787395206.110529 mapwrite 0x021e6a thru 0x02eabb (0xcc52 bytes) 000530 1787395206.193636 MIRROR_EXTEND to 2 mirrors 000531 1787395206.767251 read 0x009264 thru 0x00e6c2 (0x545f bytes) 000533 1787395206.791009 mapread 0x027550 thru 0x033c72 (0xc723 bytes) 000535 1787395206.826496 mapwrite 0x02fdee thru 0x038e2e (0x9041 bytes) 000536 1787395207.009392 read 0x0181f5 thru 0x01c385 (0x4191 bytes) 000537 1787395207.025562 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) 000538 1787395207.188835 MIRROR_EXTEND to 2 mirrors 000539 1787395207.953379 MIRROR_RESYNC 2 mirrors 000540 1787395208.625914 trunc from 0x038e2f to 0x006535 (0x328f9 bytes) 000541 1787395208.794846 mapwrite 0x00ae8a thru 0x00b77a (0x8f1 bytes) 000542 1787395208.994022 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) 000544 1787395209.214394 write 0x002d47 thru 0x005154 (0x240e bytes) 000545 1787395209.251530 trunc from 0x00b77b to 0x0313e4 (0x25c6a bytes) 000552 1787395209.314415 mapwrite 0x02223b thru 0x0308f6 (0xe6bc bytes) 000554 1787395209.480031 mapread 0x002365 thru 0x00f619 (0xd2b5 bytes) 000558 1787395209.517387 MIRROR_EXTEND to 2 mirrors 000559 1787395210.670403 MIRROR_EXTEND to 3 mirrors 000563 1787395211.380889 write 0x02e206 thru 0x039a3c (0xb837 bytes) 000564 1787395211.445922 mapread 0x019df3 thru 0x028745 (0xe953 bytes) 000565 1787395211.498156 read 0x00cf90 thru 0x016f69 (0x9fda bytes) 000566 1787395211.505616 mapread 0x02e5fd thru 0x039a3c (0xb440 bytes) 000567 1787395211.507903 write 0x02ac12 thru 0x0304f3 (0x58e2 bytes) 000569 1787395211.510205 mapread 0x027b97 thru 0x027c7b (0xe5 bytes) 000570 1787395211.511341 trunc from 0x039a3d to 0x01a88a (0x1f1b2 bytes) 000571 1787395211.536741 mapread 0x001f89 thru 0x00596c (0x39e4 bytes) 000572 1787395211.547399 write 0x002297 thru 0x010213 (0xdf7d bytes) 000573 1787395211.549066 trunc from 0x01a88b to 0x03e884 (0x23ffa bytes) 000574 1787395211.572116 mapread 0x02dbba thru 0x03ce0d (0xf254 bytes) 000575 1787395211.589318 mapwrite 0x0150b6 thru 0x01fbf0 (0xab3b bytes) 000577 1787395211.644031 read 0x015c9d thru 0x0220b5 (0xc419 bytes) 000578 1787395211.644718 mapwrite 0x0257af thru 0x02f758 (0x9faa bytes) 000579 1787395211.735088 write 0x00eec4 thru 0x0107ff (0x193c bytes) 000580 1787395211.739493 trunc from 0x03e885 to 0x004dd3 (0x39ab1 bytes) 000582 1787395211.754819 read 0x001448 thru 0x00427e (0x2e37 bytes) 000583 1787395211.755294 mapread 0x001552 thru 0x004dd3 (0x3882 bytes) 000585 1787395211.756954 mapread 0x002cd5 thru 0x004dd3 (0x20ff bytes) 000586 1787395211.760855 read 0x002f5f thru 0x004dd3 (0x1e75 bytes) 000587 1787395211.761499 write 0x01b493 thru 0x01db55 (0x26c3 bytes) 000590 1787395211.763020 read 0x010a98 thru 0x018f73 (0x84dc bytes) 000591 1787395211.769101 mapwrite 0x01e60e thru 0x02db35 (0xf528 bytes) 000593 1787395211.912209 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2db36): Mirror 3: 0xa3868532 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2db36): Mirror 3: 0xa3868532 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2db36: skipped 000599 1787395212.276362 mapwrite 0x0172bf thru 0x01e130 (0x6e72 bytes) 000600 1787395212.348106 write 0x03d11b thru 0x03ffff (0x2ee5 bytes) 000602 1787395212.349386 mapread 0x005f36 thru 0x00b4db (0x55a6 bytes) 000603 1787395212.368356 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x95fe82da (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb1c13c9e Mirror 4: 0xb1c13c9e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000604 1787395213.312635 trunc from 0x040000 to 0x012948 (0x2d6b7 bytes) 000605 1787395213.494586 mapread 0x00cbd1 thru 0x012948 (0x5d78 bytes) 000606 1787395213.520717 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x12949): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12949): Mirror 3: 0x606e74ad (stale) Mirror 4: 0x25830377 (stale) CRC-32 checksum value for chunk [0, 0x12949): Mirror 1: 0x25830377 Mirror 5: 0x25830377 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12949: skipped 000607 1787395214.606669 trunc from 0x012949 to 0x03cb24 (0x2a1dc bytes) 000608 1787395214.711433 mapwrite 0x03175f thru 0x038015 (0x68b7 bytes) 000610 1787395214.805028 read 0x0386d9 thru 0x03cb24 (0x444c bytes) 000611 1787395214.815149 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cb25): Mirror 3: 0x244387be (stale) Mirror 5: 0xd3d9ba9c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cb25): Mirror 3: 0x244387be (stale) Mirror 5: 0xd3d9ba9c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cb25: skipped 000612 1787395215.244942 mapwrite 0x02874c thru 0x037f9d (0xf852 bytes) 000614 1787395215.370381 mapwrite 0x00dd57 thru 0x019918 (0xbbc2 bytes) 000615 1787395215.469903 mapread 0x026158 thru 0x028455 (0x22fe bytes) 000616 1787395215.472741 mapread 0x0247e2 thru 0x02f72f (0xaf4e bytes) 000617 1787395215.474973 trunc from 0x03cb25 to 0x03dc15 (0x10f1 bytes) 000618 1787395215.497645 write 0x002b01 thru 0x007573 (0x4a73 bytes) 000620 1787395215.508073 write 0x01f68f thru 0x022ed7 (0x3849 bytes) 000621 1787395215.519911 write 0x01a9fa thru 0x020556 (0x5b5d bytes) 000627 1787395215.527232 mapwrite 0x0167c4 thru 0x017433 (0xc70 bytes) 000628 1787395215.594943 mapread 0x02661a thru 0x0283b7 (0x1d9e bytes) 000629 1787395215.596709 mapwrite 0x0143b2 thru 0x0210bf (0xcd0e bytes) 000631 1787395215.661434 mapwrite 0x020a08 thru 0x029c70 (0x9269 bytes) 000632 1787395215.746853 trunc from 0x03dc16 to 0x02a640 (0x135d5 bytes) 000635 1787395215.776344 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a641): Mirror 5: 0xba935780 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a641): Mirror 5: 0xba935780 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a641: skipped 000637 1787395216.128221 mapwrite 0x014f42 thru 0x015a88 (0xb47 bytes) 000638 1787395216.223319 write 0x02a52e thru 0x02d188 (0x2c5b bytes) 000639 1787395216.235304 trunc from 0x02d189 to 0x0046f3 (0x28a95 bytes) 000640 1787395216.269541 MIRROR_RESYNC 2 mirrors 000643 1787395216.887277 mapwrite 0x02343e thru 0x02face (0xc691 bytes) 000644 1787395217.072689 mapwrite 0x01d3f1 thru 0x02a964 (0xd574 bytes) 000645 1787395217.142311 mapwrite 0x02318a thru 0x028bbe (0x5a35 bytes) 000647 1787395217.199575 mapread 0x01e389 thru 0x02aa1d (0xc695 bytes) 000648 1787395217.201458 trunc from 0x02facf to 0x00039b (0x2f733 bytes) 000649 1787395217.227898 trunc from 0x00039c to 0x034512 (0x34177 bytes) 000650 1787395217.246981 mapwrite 0x003c22 thru 0x0117eb (0xdbca bytes) 000651 1787395217.334232 mapread 0x02ba9e thru 0x02ddf8 (0x235b bytes) 000653 1787395217.348902 read 0x0161d1 thru 0x025106 (0xef36 bytes) 000655 1787395217.362177 mapread 0x025344 thru 0x026981 (0x163e bytes) 000657 1787395217.364249 read 0x019d1b thru 0x01e269 (0x454f bytes) 000658 1787395217.365507 mapread 0x02b546 thru 0x034512 (0x8fcd bytes) 000659 1787395217.381624 mapread 0x029dec thru 0x02e3e6 (0x45fb bytes) 000660 1787395217.384127 read 0x02472c thru 0x0267b2 (0x2087 bytes) 000662 1787395217.386003 write 0x03f134 thru 0x03ffff (0xecc bytes) 000663 1787395217.388211 mapread 0x0193f6 thru 0x025912 (0xc51d bytes) 000666 1787395217.393104 mapwrite 0x009a13 thru 0x011049 (0x7637 bytes) 000667 1787395217.465048 write 0x005ef7 thru 0x00f426 (0x9530 bytes) 000669 1787395217.472814 MIRROR_RESYNC 2 mirrors 000670 1787395218.147140 trunc from 0x040000 to 0x000250 (0x3fdaf bytes) 000671 1787395218.257669 mapread 0x000197 thru 0x000250 (0xba bytes) 000672 1787395218.268433 read 0x00002c thru 0x000250 (0x225 bytes) 000674 1787395218.270457 read 0x000002 thru 0x000250 (0x24f bytes) 000676 1787395218.272643 mapread 0x000222 thru 0x000250 (0x2f bytes) 000678 1787395218.274666 trunc from 0x000251 to 0x0095f5 (0x93a5 bytes) 000679 1787395218.298971 read 0x004459 thru 0x0095f5 (0x519d bytes) 000680 1787395218.309432 write 0x033b4c thru 0x03b49f (0x7954 bytes) 000681 1787395218.315303 MIRROR_RESYNC 2 mirrors 000684 1787395219.438894 mapwrite 0x017102 thru 0x0231d6 (0xc0d5 bytes) 000685 1787395219.749008 read 0x007fd8 thru 0x00eb8f (0x6bb8 bytes) 000686 1787395219.779109 mapread 0x0252d0 thru 0x02e3b0 (0x90e1 bytes) 000687 1787395219.809090 mapread 0x02c694 thru 0x034beb (0x8558 bytes) 000688 1787395219.826718 mapread 0x01f6e5 thru 0x0239ca (0x42e6 bytes) 000689 1787395219.828984 write 0x0366f7 thru 0x03ffff (0x9909 bytes) 000690 1787395219.856882 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) 000691 1787395220.062567 mapread 0x02ece6 thru 0x03beac (0xd1c7 bytes) 000695 1787395220.111115 mapread 0x01101c thru 0x01b852 (0xa837 bytes) 000697 1787395220.132254 read 0x020cee thru 0x028770 (0x7a83 bytes) 000698 1787395220.142730 trunc from 0x040000 to 0x004bb7 (0x3b448 bytes) 000699 1787395220.237454 trunc from 0x004bb8 to 0x01fc73 (0x1b0bc bytes) 000700 1787395220.311356 read 0x0157f3 thru 0x01fc73 (0xa481 bytes) 000701 1787395220.348155 trunc from 0x01fc74 to 0x009cae (0x15fc5 bytes) 000702 1787395220.406113 trunc from 0x009caf to 0x028ae4 (0x1ee36 bytes) 000703 1787395220.422658 trunc from 0x028ae5 to 0x0267bf (0x2325 bytes) 000704 1787395220.458257 mapwrite 0x0251f2 thru 0x02be5c (0x6c6b bytes) 000706 1787395220.549234 mapwrite 0x01d7cf thru 0x01f7f2 (0x2024 bytes) 000708 1787395220.618216 read 0x0175e8 thru 0x020f7b (0x9994 bytes) 000711 1787395220.645530 MIRROR_EXTEND to 2 mirrors 000712 1787395221.760044 MIRROR_EXTEND to 3 mirrors 000713 1787395222.683808 mapread 0x003646 thru 0x0130de (0xfa99 bytes) 000714 1787395222.762294 mapread 0x00687c thru 0x010abf (0xa244 bytes) 000715 1787395222.783139 read 0x00b3d0 thru 0x017740 (0xc371 bytes) 000717 1787395222.804959 mapwrite 0x020637 thru 0x0253f0 (0x4dba bytes) 000719 1787395223.036623 trunc from 0x02be5d to 0x02bcac (0x1b0 bytes) 000721 1787395223.093632 trunc from 0x02bcad to 0x02bed9 (0x22d bytes) 000723 1787395223.120833 mapwrite 0x039e3d thru 0x03ffff (0x61c3 bytes) 000724 1787395223.317488 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x13e18211 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x13e18211 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000726 1787395223.761635 write 0x038ba5 thru 0x03ffff (0x745b bytes) 000729 1787395223.785974 mapwrite 0x001faa thru 0x011065 (0xf0bc bytes) 000732 1787395223.999533 MIRROR_RESYNC 2 mirrors 000733 1787395225.055043 write 0x0340e4 thru 0x0372b5 (0x31d2 bytes) 000734 1787395225.273006 read 0x00fc6b thru 0x011ddf (0x2175 bytes) truncating to largest ever: 0x3ff51 000736 1787395225.281930 trunc from 0x040000 to 0x03ff50 (0xaf bytes) 000737 1787395225.320665 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) 000738 1787395225.642896 mapread 0x017be6 thru 0x020aec (0x8f07 bytes) 000739 1787395225.691392 read 0x0350fe thru 0x03e8d8 (0x97db bytes) 000740 1787395225.713523 read 0x005478 thru 0x007e98 (0x2a21 bytes) 000743 1787395225.729366 trunc from 0x03ff51 to 0x020ca1 (0x1f2af bytes) 000744 1787395225.745627 write 0x026b20 thru 0x035274 (0xe755 bytes) 000745 1787395225.748013 trunc from 0x035275 to 0x02c1cc (0x90a8 bytes) 000746 1787395225.769950 write 0x03870f thru 0x03ffff (0x78f1 bytes) 000747 1787395225.771614 trunc from 0x040000 to 0x00ea73 (0x3158c bytes) 000748 1787395225.813174 MIRROR_EXTEND to 2 mirrors 000750 1787395227.006249 mapread 0x003628 thru 0x00b5bd (0x7f96 bytes) 000751 1787395227.047084 trunc from 0x00ea74 to 0x02f445 (0x209d2 bytes) 000752 1787395227.137553 read 0x00f741 thru 0x0105c0 (0xe80 bytes) 000753 1787395227.160217 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) 000757 1787395227.334132 trunc from 0x02f446 to 0x02d764 (0x1ce1 bytes) 000761 1787395227.362160 mapread 0x004f08 thru 0x0127fb (0xd8f4 bytes) 000764 1787395227.406208 mapread 0x01d667 thru 0x0244b8 (0x6e52 bytes) 000766 1787395227.415903 trunc from 0x02d765 to 0x0031e2 (0x2a582 bytes) 000769 1787395227.444403 write 0x01bf96 thru 0x0288a1 (0xc90c bytes) 000771 1787395227.447528 mapread 0x020ebd thru 0x0288a1 (0x79e5 bytes) 000773 1787395227.455291 MIRROR_EXTEND to 2 mirrors 000777 1787395228.421417 write 0x0053e4 thru 0x00fd71 (0xa98e bytes) 000778 1787395228.542211 write 0x024043 thru 0x028982 (0x4940 bytes) 000779 1787395228.553919 write 0x00a31e thru 0x00b840 (0x1523 bytes) 000781 1787395228.556547 mapread 0x018172 thru 0x0262d4 (0xe163 bytes) 000783 1787395228.617285 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x28983): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28983): Mirror 2: 0x5966d86d (stale) CRC-32 checksum value for chunk [0, 0x28983): Mirror 1: 0xdd1b2f28 Mirror 3: 0xdd1b2f28 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28983: skipped 000787 1787395230.234633 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x28983): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28983): Mirror 2: 0x5966d86d (stale) CRC-32 checksum value for chunk [0, 0x28983): Mirror 1: 0xdd1b2f28 Mirror 3: 0xdd1b2f28 Mirror 4: 0xdd1b2f28 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28983: skipped 000789 1787395231.551490 trunc from 0x028983 to 0x00c6a6 (0x1c2dc bytes) 000791 1787395231.658515 MIRROR_RESYNC 4 mirrors 000793 1787395233.233623 trunc from 0x00c6a7 to 0x02b2ab (0x1ec05 bytes) 000794 1787395233.347416 write 0x010b3e thru 0x01f6cc (0xeb8f bytes) 000795 1787395233.373464 mapwrite 0x01f8ec thru 0x02167a (0x1d8f bytes) 000796 1787395233.446420 trunc from 0x02b2ac to 0x027696 (0x3c15 bytes) 000797 1787395233.477187 MIRROR_RESYNC 4 mirrors 000799 1787395234.618168 read 0x020558 thru 0x0254e7 (0x4f90 bytes) 000801 1787395234.635039 trunc from 0x027697 to 0x001164 (0x26532 bytes) 000802 1787395234.786291 trunc from 0x001165 to 0x017bae (0x16a4a bytes) 000803 1787395234.854883 trunc from 0x017baf to 0x014d5a (0x2e54 bytes) 000804 1787395234.889193 trunc from 0x014d5b to 0x007095 (0xdcc5 bytes) 000807 1787395234.930265 mapread 0x005dd1 thru 0x006478 (0x6a8 bytes) 000808 1787395234.959272 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x7096): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7096): Mirror 2: 0x41ebe19a (stale) Mirror 3: 0x41ebe19a (stale) Mirror 4: 0x41ebe19a (stale) CRC-32 checksum value for chunk [0, 0x7096): Mirror 1: 0x9be1b8b9 Mirror 5: 0x9be1b8b9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7096: skipped 000809 1787395236.160301 write 0x007259 thru 0x00c44a (0x51f2 bytes) 000810 1787395236.249432 write 0x01fb10 thru 0x02c5a8 (0xca99 bytes) 000811 1787395236.254442 mapread 0x023252 thru 0x0267c1 (0x3570 bytes) 000812 1787395236.261925 MIRROR_RESYNC 5 mirrors 000813 1787395237.674464 mapwrite 0x0158bb thru 0x017796 (0x1edc bytes) 000818 1787395238.089074 mapread 0x00500d thru 0x008aff (0x3af3 bytes) 000819 1787395238.130804 mapread 0x01f864 thru 0x020bb4 (0x1351 bytes) 000822 1787395238.152396 mapread 0x0002fc thru 0x00441b (0x4120 bytes) 000823 1787395238.174925 MIRROR_RESYNC 5 mirrors 000824 1787395239.829653 mapwrite 0x02180c thru 0x028d88 (0x757d bytes) 000825 1787395240.082501 MIRROR_RESYNC 5 mirrors 000828 1787395241.531631 read 0x00925b thru 0x01127d (0x8023 bytes) 000829 1787395241.548993 mapread 0x0244ed thru 0x02a567 (0x607b bytes) 000830 1787395241.602213 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) 000832 1787395242.374273 MIRROR_EXTEND to 5 mirrors 000833 1787395243.771447 mapread 0x02397d thru 0x02c5a8 (0x8c2c bytes) 000835 1787395243.802338 mapwrite 0x00f020 thru 0x01c829 (0xd80a bytes) 000836 1787395244.071852 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c5a9): Mirror 2: 0x17d5f4fc (stale) Mirror 3: 0x17d5f4fc (stale) Mirror 5: 0x17d5f4fc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c5a9): Mirror 2: 0x17d5f4fc (stale) Mirror 3: 0x17d5f4fc (stale) Mirror 5: 0x17d5f4fc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c5a9: skipped 000837 1787395244.440311 write 0x016df4 thru 0x01a0d9 (0x32e6 bytes) 000838 1787395244.453478 trunc from 0x02c5a9 to 0x020278 (0xc330 bytes) 000839 1787395244.476487 read 0x00ccb5 thru 0x01247f (0x57cb bytes) 000842 1787395244.496322 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x20279): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20279): Mirror 2: 0xa5e01945 (stale) Mirror 3: 0xa5e01945 (stale) Mirror 5: 0xa5e01945 (stale) CRC-32 checksum value for chunk [0, 0x20279): Mirror 1: 0x4ee44808 Mirror 6: 0x4ee44808 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20279: skipped 000843 1787395245.562816 write 0x01d7ae thru 0x022628 (0x4e7b bytes) 000844 1787395245.898737 write 0x0373cc thru 0x03aaa4 (0x36d9 bytes) 000846 1787395245.900467 mapwrite 0x01d9de thru 0x0282df (0xa902 bytes) 000848 1787395246.002884 mapwrite 0x0240df thru 0x027fa8 (0x3eca bytes) 000852 1787395246.129992 mapread 0x023806 thru 0x02779d (0x3f98 bytes) 000855 1787395246.144757 write 0x00fe62 thru 0x01847a (0x8619 bytes) 000857 1787395246.178263 mapread 0x02cc63 thru 0x0370b3 (0xa451 bytes) 000858 1787395246.200198 mapread 0x00a950 thru 0x015bfb (0xb2ac bytes) 000861 1787395246.215916 write 0x032581 thru 0x03ffff (0xda7f bytes) 000862 1787395246.225074 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x15989777 (stale) Mirror 3: 0x15989777 (stale) Mirror 5: 0x15989777 (stale) Mirror 6: 0x9593966c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x88b9384c Mirror 7: 0x88b9384c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000864 1787395247.612107 read 0x02e448 thru 0x03517b (0x6d34 bytes) 000867 1787395247.679065 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x15989777 (stale) Mirror 3: 0x15989777 (stale) Mirror 5: 0x15989777 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x15989777 (stale) Mirror 3: 0x15989777 (stale) Mirror 5: 0x15989777 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x88b9384c Mirror 7: 0x88b9384c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000868 1787395248.481035 mapwrite 0x020956 thru 0x022a43 (0x20ee bytes) 000869 1787395248.793152 mapread 0x007233 thru 0x00e40c (0x71da bytes) 000870 1787395248.804839 mapwrite 0x0297f0 thru 0x02bb34 (0x2345 bytes) 000873 1787395248.881888 mapwrite 0x0352dc thru 0x03d505 (0x822a bytes) 000876 1787395248.999381 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x15989777 (stale) Mirror 3: 0x15989777 (stale) Mirror 5: 0x15989777 (stale) Mirror 7: 0x88b9384c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x73ca2717 Mirror 8: 0x73ca2717 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000878 1787395250.225551 trunc from 0x040000 to 0x01ceb2 (0x2314d bytes) 000879 1787395250.360537 mapwrite 0x00ddaf thru 0x01974c (0xb99e bytes) 000880 1787395250.550398 write 0x0112fa thru 0x01e125 (0xce2c bytes) 000881 1787395250.571062 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1e126): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e126): Mirror 2: 0x31b3a1c9 (stale) Mirror 3: 0x31b3a1c9 (stale) Mirror 5: 0x31b3a1c9 (stale) Mirror 7: 0x270e7c2d (stale) Mirror 8: 0x270e7c2d (stale) CRC-32 checksum value for chunk [0, 0x1e126): Mirror 1: 0x167f0234 Mirror 9: 0x167f0234 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e126: skipped 000883 1787395252.034368 mapread 0x01caf2 thru 0x01e125 (0x1634 bytes) 000885 1787395252.096801 mapwrite 0x00cabd thru 0x01a8fb (0xde3f bytes) 000887 1787395252.552110 mapwrite 0x02a4ee thru 0x02eca3 (0x47b6 bytes) 000888 1787395252.790212 read 0x012534 thru 0x01cb8c (0xa659 bytes) 000889 1787395252.794258 mapwrite 0x00fb7a thru 0x013df9 (0x4280 bytes) 000890 1787395252.910010 mapwrite 0x034a03 thru 0x03e47b (0x9a79 bytes) 000894 1787395253.081922 MIRROR_RESYNC 7 mirrors 000895 1787395255.248846 mapwrite 0x03396d thru 0x036376 (0x2a0a bytes) 000896 1787395255.654863 mapread 0x001be6 thru 0x0078fb (0x5d16 bytes) 000897 1787395255.671341 trunc from 0x03e47c to 0x022a6f (0x1ba0c bytes) 000898 1787395255.719603 write 0x01c97d thru 0x022061 (0x56e5 bytes) 000899 1787395255.742129 trunc from 0x022a70 to 0x012e5f (0xfc10 bytes) 000900 1787395255.769150 mapwrite 0x0279d6 thru 0x02dd32 (0x635d bytes) 000901 1787395255.911668 read 0x00a556 thru 0x00e710 (0x41bb bytes) 000902 1787395255.912476 mapread 0x0249eb thru 0x02b50e (0x6b24 bytes) 000903 1787395255.913766 trunc from 0x02dd33 to 0x00b92a (0x22408 bytes) 000904 1787395255.972974 mapread 0x00b89d thru 0x00b92a (0x8e bytes) 000905 1787395255.995861 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xb92b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb92b): Mirror 2: 0xd7ca27b6 (stale) Mirror 3: 0xd7ca27b6 (stale) Mirror 5: 0xd7ca27b6 (stale) Mirror 7: 0xd7ca27b6 (stale) Mirror 8: 0xd7ca27b6 (stale) Mirror 9: 0xd7ca27b6 (stale) CRC-32 checksum value for chunk [0, 0xb92b): Mirror 1: 0xd7ca27b6 Mirror 10: 0xd7ca27b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb92b: skipped 000906 1787395257.653652 trunc from 0x00b92b to 0x01b6e1 (0xfdb7 bytes) 000907 1787395257.806174 mapread 0x00c72e thru 0x016ddd (0xa6b0 bytes) 000908 1787395257.849286 trunc from 0x01b6e2 to 0x00b7c9 (0xff18 bytes) 000909 1787395257.894818 write 0x03d0fd thru 0x03ffff (0x2f03 bytes) 000911 1787395257.896587 mapwrite 0x01a6c0 thru 0x02087a (0x61bb bytes) 000912 1787395258.027464 MIRROR_RESYNC 8 mirrors 000915 1787395260.424436 write 0x018fc7 thru 0x022520 (0x955a bytes) 000916 1787395260.541884 MIRROR_RESYNC 8 mirrors 000918 1787395262.582081 read 0x03b5eb thru 0x03ffff (0x4a15 bytes) 000920 1787395262.601240 MIRROR_RESYNC 8 mirrors 000922 1787395263.259294 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) 000927 1787395263.783935 read 0x01eb5d thru 0x02a8ef (0xbd93 bytes) 000930 1787395263.802179 mapwrite 0x008398 thru 0x01309a (0xad03 bytes) 000931 1787395263.996818 trunc from 0x040000 to 0x008fa4 (0x3705b bytes) 000932 1787395264.029640 read 0x008ded thru 0x008fa4 (0x1b8 bytes) 000933 1787395264.030993 mapread 0x004444 thru 0x005513 (0x10d0 bytes) 000935 1787395264.035228 read 0x006775 thru 0x008fa4 (0x2830 bytes) 000937 1787395264.038293 mapwrite 0x0004f6 thru 0x0091d4 (0x8cdf bytes) 000938 1787395264.141367 write 0x006b41 thru 0x00dee9 (0x73a9 bytes) 000942 1787395264.145902 mapwrite 0x0152c4 thru 0x015c87 (0x9c4 bytes) 000943 1787395264.222180 mapread 0x00f987 thru 0x015c87 (0x6301 bytes) 000944 1787395264.225443 mapwrite 0x024e23 thru 0x029749 (0x4927 bytes) 000945 1787395264.456129 mapread 0x014ca4 thru 0x01f3a7 (0xa704 bytes) 000949 1787395264.462804 write 0x02747c thru 0x02cf6d (0x5af2 bytes) 000951 1787395264.466027 write 0x0330b5 thru 0x03e258 (0xb1a4 bytes) 000955 1787395264.472184 write 0x015823 thru 0x015db9 (0x597 bytes) 000956 1787395264.477528 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e259): Mirror 2: 0x9c6435d1 (stale) Mirror 5: 0x9c6435d1 (stale) Mirror 7: 0x9c6435d1 (stale) Mirror 9: 0x9c6435d1 (stale) Mirror 10: 0x9c6435d1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e259): Mirror 2: 0x9c6435d1 (stale) Mirror 5: 0x9c6435d1 (stale) Mirror 7: 0x9c6435d1 (stale) Mirror 9: 0x9c6435d1 (stale) Mirror 10: 0x9c6435d1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e259: skipped 000957 1787395265.207961 write 0x02261b thru 0x02d792 (0xb178 bytes) 000959 1787395265.235610 read 0x006f5b thru 0x016b7e (0xfc24 bytes) 000961 1787395265.253428 write 0x006ffa thru 0x009f76 (0x2f7d bytes) 000962 1787395265.254602 read 0x00fa66 thru 0x017ef4 (0x848f bytes) 000963 1787395265.276604 MIRROR_RESYNC 6 mirrors 000965 1787395267.163367 write 0x01a695 thru 0x021fe0 (0x794c bytes) 000967 1787395267.352313 read 0x02c105 thru 0x0370e6 (0xafe2 bytes) 000972 1787395267.371529 MIRROR_RESYNC 6 mirrors 000973 1787395268.925538 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x169:0x0]: Invalid argument (22) 000974 1787395269.485926 trunc from 0x03e259 to 0x00d105 (0x31153 bytes) 000976 1787395269.609875 mapwrite 0x03e688 thru 0x03ec24 (0x59d bytes) 000977 1787395269.697337 read 0x020747 thru 0x02bb4f (0xb409 bytes) 000978 1787395269.721285 read 0x013e6d thru 0x01bf14 (0x80a8 bytes) 000979 1787395269.751765 mapread 0x0384fc thru 0x03ec24 (0x6729 bytes) 000981 1787395269.753404 mapwrite 0x03db7d thru 0x03e261 (0x6e5 bytes) 000982 1787395269.810375 write 0x01e4bc thru 0x025fdd (0x7b22 bytes) 000983 1787395269.813886 mapwrite 0x01bfd6 thru 0x02673f (0xa76a bytes) 000984 1787395269.890186 trunc from 0x03ec25 to 0x01688f (0x28395 bytes) 000987 1787395269.944170 trunc from 0x016890 to 0x02a4f4 (0x13c65 bytes) 000988 1787395269.991435 mapread 0x019fb6 thru 0x01f5be (0x5609 bytes) 000990 1787395270.083895 trunc from 0x02a4f5 to 0x02f72c (0x5238 bytes) 000991 1787395270.138513 MIRROR_RESYNC 5 mirrors 000992 1787395271.654209 trunc from 0x02f72d to 0x02a3bb (0x5371 bytes) 000993 1787395271.874314 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2a3bc): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a3bc): Mirror 2: 0x5dfae0bc (stale) Mirror 5: 0x5dfae0bc (stale) Mirror 9: 0x5dfae0bc (stale) Mirror 10: 0x5dfae0bc (stale) CRC-32 checksum value for chunk [0, 0x2a3bc): Mirror 1: 0x5dfae0bc Mirror 11: 0x5dfae0bc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a3bc: skipped 000994 1787395273.456066 mapread 0x013de1 thru 0x014aeb (0xd0b bytes) 000995 1787395273.489196 read 0x01953f thru 0x0262a4 (0xcd66 bytes) 000996 1787395273.511076 write 0x03ab76 thru 0x03d960 (0x2deb bytes) 000998 1787395273.606022 trunc from 0x03d961 to 0x01de65 (0x1fafb bytes) 001000 1787395273.648199 mapread 0x001808 thru 0x007c9a (0x6493 bytes) All operations completed A-OK! PASS 70a (142s) == sanity-flr test 71: check mirror extend parent fid ==== 06:41:20 (1787395280) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 06:41:25 (1787395285) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.489459 s, 4.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374233 s, 2.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a2:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a2:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29f:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 06:41:37 (1787395297) Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre2 Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 828416 .. Extending file size to 1346528 .. failed Extending file size to 6733888 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4608256 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8474400 .. Extending file size to 8697664 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8553728 .. Extending file size to 8815968 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9007360 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 56904 56905 56906 56908 56909 done 192.168.202.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg231-client.virtnet /mnt/lustre2 (opts:) 192.168.202.131@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg231-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (89s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 06:43:05 (1787395385) Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre2 Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5172736 .. Extending file size to 4726592 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7662752 .. Extending file size to 6260416 .. Extending file size to 8441888 .. Extending file size to 6498624 .. Extending file size to 7901568 .. Extending file size to 7907296 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8229088 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8995488 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8769760 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58393 58394 58396 58397 58399 done 192.168.202.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg231-client.virtnet /mnt/lustre2 (opts:) 192.168.202.131@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg231-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (83s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 06:44:29 (1787395469) Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.157317 s, 13.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.496888 s, 4.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xdd41a58 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg231-client.virtnet /mnt/lustre2 (opts:) PASS 200c (24s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 06:44:54 (1787395494) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.179402 s, 11.7 MB/s PASS 202 (10s) == sanity-flr test 203: mirror file preserve mirror ID === 06:45:04 (1787395504) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b8:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b8:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2b4:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 06:45:13 (1787395513) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.207768 s, 10.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ba:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 06:45:21 (1787395521) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.284794 s, 7.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bc:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bc:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b8:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 06:45:29 (1787395529) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.308825 s, 6.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ba:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 06:46:04 (1787395564) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg231-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134951 s, 7.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bf:0x0] } PASS 204d (65s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 06:47:09 (1787395629) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg231-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2be:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.371117 s, 5.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2be:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2be:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (67s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 06:48:16 (1787395696) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg231-server mds-ost sync done. Creating new pool test_204f oleg231-server: Pool lustre.test_204f created Adding targets to pool oleg231-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg231-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.242404 s, 8.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg231-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg231-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg231-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 06:49:36 (1787395776) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 06:49:44 (1787395784) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 06:49:51 (1787395791) oleg231-server: Pool lustre.test_206 created Adding targets to pool oleg231-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg231-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg231-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg231-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg231-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 06:50:17 (1787395817) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707959 s, 14.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347637 s, 30.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xac41e0e6 Mirror 3: 0xac41e0e6 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 06:50:30 (1787395830) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 06:50:34 (1787395834) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 06:50:37 (1787395837) Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.150589 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xc90000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1b10000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1b10000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1b10000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2220000 Read this (4 bytes): BBBB 192.168.202.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg231-client.virtnet /mnt/lustre2 (opts:) PASS 209a (41s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 06:51:18 (1787395878) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0137835 s, 594 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 06:51:27 (1787395887) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339685 s, 30.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1ea lmm_fid: [0x200000404:0x1ea:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1ea lmm_fid: [0x200000404:0x1ea:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834410 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-22 06:51:30.000000000 -0400 Modify: 2026-08-22 06:51:30.000000000 -0400 Change: 2026-08-22 06:51:31.000000000 -0400 Birth: 2026-08-22 06:51:30.000000000 -0400 PASS 210a (11s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 06:51:38 (1787395898) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 06:51:41 (1787395901) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.470859 s, 87.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0560314 s, 73.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 22 06:51 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ec:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 22 06:51 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 06:51:51 (1787395911) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 06:52:01 (1787395921) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (13s) == sanity-flr test complete, duration 1926 sec =========== 06:52:13 (1787395933) === sanity-flr: start cleanup 06:52:15 (1787395935) === === sanity-flr: finish cleanup 06:52:22 (1787395942) ===