-----============= acceptance-small: sanity-flr ============----- Fri Jan 30 01:33:44 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg350-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 01:34:07 (1769754847) === oleg350-client.virtnet: executing check_config_client /mnt/lustre oleg350-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg350-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8c7a42aa2000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c7a42aa2000.idle_timeout=debug disable quota as required oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 01:34:48 (1769754888) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8185] == sanity-flr test 0a: lfs mirror create with -N option == 01:34:50 (1769754890) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:35:07 (1769754907) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:35:11 (1769754911) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:35:15 (1769754915) PASS 0d (21s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:35:36 (1769754936) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:35:40 (1769754940) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 01:35:43 (1769754943) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8c7a42aa2000.stats=clear osc.lustre-OST0001-osc-ffff8c7a42aa2000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:35:51 (1769754951) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:36:04 (1769754964) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:36:12 (1769754972) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:36:16 (1769754976) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (7s) == sanity-flr test 2: create components from existing files ========================================================== 01:36:23 (1769754983) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:36:31 (1769754991) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:36:39 (1769754999) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:36:47 (1769755007) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:37:01 (1769755021) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:37:08 (1769755028) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664697 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094362 s, 11.1 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:37:16 (1769755036) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 0.851514 s, 20.9 MB/s PASS 22 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:37:25 (1769755045) Stopping /mnt/lustre-ost1 (opts:) on oleg350-server oleg350-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c7a42aa2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c7a42aa2000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg350-server: oleg350-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg350-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg350-server oleg350-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c7a42aa2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c7a42aa2000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg350-server: oleg350-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg350-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (68s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:38:33 (1769755113) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.285828 s, 14.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg350-server oleg350-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c7a42aa2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c7a42aa2000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg350-server: oleg350-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg350-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:39:17 (1769755157) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg350-server oleg350-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c7a42aa2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c7a42aa2000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg350-server: oleg350-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg350-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg350-server oleg350-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c7a42aa2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c7a42aa2000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg350-server: oleg350-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg350-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (74s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:40:32 (1769755232) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.01951 s, 17.4 MB/s -rw-r--r-- 1 root root 52428800 Jan 30 01:40 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.925s user 0m0.000s sys 0m0.164s Stopping /mnt/lustre-ost1 (opts:) on oleg350-server oleg350-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c7a42aa2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c7a42aa2000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.944s user 0m0.000s sys 0m0.155s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg350-server: oleg350-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg350-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg350-server oleg350-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c7a42aa2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c7a42aa2000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.305s user 0m0.000s sys 0m0.169s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg350-server: oleg350-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg350-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg350-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c7a42aa2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c7a42aa2000.ost_server_uuid in FULL state after 0 sec PASS 33b (83s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:41:55 (1769755315) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:41:58 (1769755318) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:42:01 (1769755321) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 01:42:03 (1769755323) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 01:42:10 (1769755330) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.392381 s, 32.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:42:21 (1769755341) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457243 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591791 s, 17.7 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:42:28 (1769755348) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:42:37 (1769755357) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00179324 s, 571 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00165439 s, 619 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00123569 s, 829 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00069553 s, 1.5 MB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 01:42:47 (1769755367) create test files with size 15745924 .. a0c7cbcc1ab96d71b8df9594521d37cc - 80f17404c062802cf69c962fe048fe36 - cf20d2032c769fc5185fcc18311f62d2 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg350-client.virtnet /mnt/lustre (opts:) Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 01:43:11 (1769755391) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:43:26 (1769755406) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:43:33 (1769755413) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820955 s, 12.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022749 s, 46.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 01:43:42 (1769755422) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0920275 s, 22.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.345417 s, 12.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.136229 s, 15.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 01:43:56 (1769755436) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 01:44:00 (1769755440) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:44:03 (1769755443) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221285 s, 47.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 01:44:09 (1769755449) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.242119 s, 13.0 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:44:18 (1769755458) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x48:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x46:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x46:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:44:28 (1769755468) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.284215 s, 36.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg350-client.virtnet /mnt/lustre (opts:) Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:44:41 (1769755481) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8c7a50d98800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.351566 s, 29.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg350-client.virtnet /mnt/lustre (opts:) Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8c7a51b9b000.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:44:58 (1769755498) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (37s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:45:35 (1769755535) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (37s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:46:12 (1769755572) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 01:46:15 (1769755575) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.187017 s, 16.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.216367 s, 14.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x46f3057b (stale) Mirror 2: 0x46f3057b (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 1425874a491fcbf439f00fd89a6b67f0 - mirror 1 checksum: 6e0fb1c6be0fcd8cb3d3b3d2d2e036e2 - mirror 2 checksum: 1425874a491fcbf439f00fd89a6b67f0 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Jan 30 01:46:17 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Jan 30 01:46:17 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 1425874a491fcbf439f00fd89a6b67f0 - mirror 1 checksum: 6e0fb1c6be0fcd8cb3d3b3d2d2e036e2 - mirror 2 checksum: 6e0fb1c6be0fcd8cb3d3b3d2d2e036e2 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Jan 30 01:46:17 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Jan 30 01:46:17 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 01:46:24 (1769755584) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019821 s, 52.9 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (6s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:46:30 (1769755590) Failing mds1 on oleg350-server Stopping /mnt/lustre-mds1 (opts:) on oleg350-server 01:46:41 (1769755601) shut down facet: mds1 facet_host: oleg350-server facet_failover_host: oleg350-server Failover mds1 to oleg350-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 01:47:13 (1769755633) targets are mounted 01:47:14 (1769755634) facet_failover done oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462929 s, 22.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } PASS 50A (58s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:47:28 (1769755648) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0269434 s, 760 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0120971 s, 1.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (8s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:47:36 (1769755656) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.108283 s, 189 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0123389 s, 1.7 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:47:46 (1769755666) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00562871 s, 2.9 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:47:52 (1769755672) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0362998 s, 564 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:48:00 (1769755680) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00757716 s, 541 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0199705 s, 205 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:48:07 (1769755687) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19:0x0]: Invalid argument (22) PASS 60b (19s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:48:26 (1769755706) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:48:53 (1769755733) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:49:15 (1769755755) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:49:39 (1769755779) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 01:49:44 (1769755784) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 55919 mirror split pid 55920 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7638: 55920 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 55919 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) PASS 70 (65s) == sanity-flr test 70a: flr mode fsx test ================ 01:50:50 (1769755850) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 5851 fallocating to largest ever: 0x13eee 000001 1769755852.261566 fallocate 0x00a299 thru 0x013eed (0x9c55 bytes) 000002 1769755852.270407 mapread 0x013202 thru 0x013eed (0xcec bytes) 000003 1769755852.274970 punch from 0x005336 to 0x00bed8 (0x6ba3 bytes) truncating to largest ever: 0x3a5b4 000005 1769755852.279402 trunc from 0x013eee to 0x03a5b3 (0x266c6 bytes) 000006 1769755852.290260 fallocate 0x009331 thru 0x00cca3 (0x3973 bytes) 000007 1769755852.293842 trunc from 0x03a5b4 to 0x00b47a (0x2f139 bytes) 000009 1769755852.302999 write 0x01b814 thru 0x01de3d (0x262a bytes) 000010 1769755852.304358 fallocate 0x028f8a thru 0x02e37f (0x53f6 bytes) 000011 1769755852.307579 read 0x00b722 thru 0x00c10a (0x9e9 bytes) 000012 1769755852.318619 fallocate 0x00e35d thru 0x017139 (0x8ddd bytes) 000013 1769755852.331070 fallocate 0x00be1c thru 0x00e2a3 (0x2488 bytes) 000016 1769755852.345393 punch from 0x02a7a4 to 0x02e37f (0x3bdc bytes) 000017 1769755852.380989 MIRROR_EXTEND to 2 mirrors 000018 1769755853.153802 fallocate 0x003891 thru 0x00e3ee (0xab5e bytes) 000019 1769755853.200351 write 0x022d3f thru 0x0308a8 (0xdb6a bytes) 000020 1769755853.214369 mapread 0x010e95 thru 0x01232e (0x149a bytes) 000021 1769755853.229603 mapwrite 0x03dd4d thru 0x03ffff (0x22b3 bytes) 000022 1769755853.294347 trunc from 0x040000 to 0x036843 (0x97bc bytes) 000023 1769755853.311222 read 0x01d45e thru 0x024399 (0x6f3c bytes) 000024 1769755853.329240 mapread 0x00b137 thru 0x01686b (0xb735 bytes) 000025 1769755853.337394 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4f:0x0]: Invalid argument (22) 000026 1769755853.516967 trunc from 0x036844 to 0x01421f (0x22624 bytes) 000027 1769755853.548063 trunc from 0x0040a7 to 0x0081c0 (0x411a bytes) 000028 1769755853.553974 mapread 0x004360 thru 0x0119ea (0xd68b bytes) 000029 1769755853.589163 write 0x00d94b thru 0x01b8d6 (0xdf8c bytes) 000032 1769755853.601604 read 0x0020d7 thru 0x00dd5e (0xbc88 bytes) 000033 1769755853.602314 punch from 0x00f159 to 0x016f21 (0x7dc9 bytes) 000034 1769755853.636137 read 0x00cc0a thru 0x01b8d6 (0xeccd bytes) 000035 1769755853.646253 read 0x00a27a thru 0x00acf4 (0xa7b bytes) 000036 1769755853.648335 trunc from 0x01000b to 0x013de8 (0x3dde bytes) 000037 1769755853.674350 mapwrite 0x00059b thru 0x001406 (0xe6c bytes) 000038 1769755853.713926 read 0x00101c thru 0x009649 (0x862e bytes) 000039 1769755853.716740 write 0x02c048 thru 0x03556f (0x9528 bytes) 000040 1769755853.721339 mapread 0x01394e thru 0x01596e (0x2021 bytes) 000041 1769755853.745436 trunc from 0x01281b to 0x01ee73 (0xc659 bytes) 000043 1769755853.754087 trunc from 0x035570 to 0x017c29 (0x1d946 bytes) 000044 1769755853.804336 write 0x026da7 thru 0x0349db (0xdc35 bytes) 000045 1769755853.806522 trunc from 0x01baad to 0x01bf1e (0x472 bytes) 000046 1769755853.813327 write 0x0032bd thru 0x00fdd3 (0xcb17 bytes) truncating to largest ever: 0x3fcb5 000047 1769755853.816432 trunc from 0x0349dc to 0x03fcb4 (0xb2d9 bytes) 000048 1769755853.844433 read 0x028939 thru 0x0320ff (0x97c7 bytes) 000049 1769755853.845795 fallocate 0x00f39b thru 0x011b62 (0x27c8 bytes) 000050 1769755853.852333 punch from 0x028ea9 to 0x031ad7 (0x8c2f bytes) 000051 1769755853.875530 trunc from 0x0085a1 to 0x00e783 (0x61e3 bytes) 000052 1769755853.882473 mapwrite 0x0223e9 thru 0x02e46f (0xc087 bytes) 000053 1769755853.904583 read 0x036aa9 thru 0x03fcb4 (0x920c bytes) 000054 1769755853.905390 mapwrite 0x03e355 thru 0x03ffff (0x1cab bytes) fallocating to largest ever: 0x40000 000055 1769755853.927927 fallocate 0x03e6c0 thru 0x03ffff (0x1940 bytes) 000057 1769755853.934569 fallocate 0x005cfc thru 0x0116c6 (0xb9cb bytes) 000059 1769755853.949882 trunc from 0x021093 to 0x02830b (0x7279 bytes) 000060 1769755853.955342 trunc from 0x040000 to 0x03c8ee (0x3711 bytes) 000062 1769755853.970849 fallocate 0x024051 thru 0x0328c8 (0xe878 bytes) 000063 1769755853.975626 read 0x005a75 thru 0x00d73d (0x7cc9 bytes) 000064 1769755853.976591 write 0x008c0e thru 0x011999 (0x8d8c bytes) 000065 1769755853.978262 trunc from 0x03c8ef to 0x035aac (0x6e42 bytes) 000066 1769755853.997872 punch from 0x02923d to 0x02fa17 (0x67db bytes) 000067 1769755854.013653 mapread 0x01369a thru 0x0214e3 (0xde4a bytes) 000068 1769755854.036508 trunc from 0x035aad to 0x02ee65 (0x6c47 bytes) 000069 1769755854.064508 MIRROR_EXTEND to 2 mirrors 000070 1769755854.821983 mapread 0x01c74b thru 0x02aa16 (0xe2cc bytes) 000072 1769755854.893329 mapwrite 0x00e5ca thru 0x01741a (0x8e51 bytes) 000073 1769755855.034407 mapwrite 0x003a91 thru 0x00b914 (0x7e84 bytes) 000074 1769755855.068687 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2ee66): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ee66): Mirror 2: 0xe6f8c73 (stale) CRC-32 checksum value for chunk [0, 0x2ee66): Mirror 1: 0xf12e49ca Mirror 3: 0xf12e49ca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ee66: skipped 000075 1769755855.831189 mapwrite 0x022458 thru 0x02f744 (0xd2ed bytes) 000076 1769755856.018971 punch from 0x00e4f7 to 0x00e79e (0x2a8 bytes) 000077 1769755856.031123 MIRROR_RESYNC 3 mirrors 000078 1769755856.654670 MIRROR_EXTEND to 4 mirrors 000079 1769755857.072288 trunc from 0x02f745 to 0x016d96 (0x189ae bytes) 000080 1769755857.105909 mapwrite 0x01783f thru 0x01b863 (0x4025 bytes) 000081 1769755857.148207 read 0x001c7a thru 0x010772 (0xeaf9 bytes) 000082 1769755857.158078 mapwrite 0x001977 thru 0x003be2 (0x226c bytes) 000083 1769755857.169955 mapread 0x017285 thru 0x01776d (0x4e9 bytes) 000084 1769755857.171294 read 0x00d773 thru 0x01105d (0x38eb bytes) 000085 1769755857.178981 punch from 0x0173b4 to 0x01b863 (0x44b0 bytes) 000086 1769755857.184190 fallocate 0x024529 thru 0x02ba80 (0x7558 bytes) 000088 1769755857.194314 MIRROR_RESYNC 4 mirrors 000089 1769755858.112625 write 0x038320 thru 0x03e288 (0x5f69 bytes) 000090 1769755858.161543 trunc from 0x018cf5 to 0x019e48 (0x1154 bytes) 000091 1769755858.167280 punch from 0x02e25d to 0x0367b3 (0x8557 bytes) 000092 1769755858.172193 trunc from 0x03e289 to 0x000070 (0x3e218 bytes) 000093 1769755858.229211 mapwrite 0x00b112 thru 0x011878 (0x6767 bytes) 000094 1769755858.311509 write 0x01de62 thru 0x0253b3 (0x7552 bytes) 000095 1769755858.313695 fallocate 0x00743b thru 0x0111a5 (0x9d6b bytes) 000096 1769755858.346728 punch from 0x001769 to 0x00dcae (0xc546 bytes) 000097 1769755858.354565 trunc from 0x0253b4 to 0x036ec6 (0x11b13 bytes) 000098 1769755858.375548 write 0x005f25 thru 0x00f36b (0x9447 bytes) 000099 1769755858.384355 trunc from 0x036ec7 to 0x03bfaa (0x50e4 bytes) 000100 1769755858.428841 MIRROR_RESYNC 4 mirrors 000101 1769755859.694787 punch from 0x019382 to 0x02493d (0xb5bc bytes) 000102 1769755859.735108 mapwrite 0x02cfbf thru 0x038f98 (0xbfda bytes) 000103 1769755859.792359 mapwrite 0x031919 thru 0x03ee4c (0xd534 bytes) 000104 1769755859.865252 read 0x035c35 thru 0x03ee4c (0x9218 bytes) 000105 1769755859.868705 trunc from 0x03ee4d to 0x02b236 (0x13c16 bytes) 000106 1769755859.891443 read 0x01f839 thru 0x02a142 (0xa90a bytes) 000107 1769755859.907805 fallocate 0x01d952 thru 0x023c69 (0x6318 bytes) 000108 1769755859.913442 punch from 0x013908 to 0x021936 (0xe02f bytes) 000109 1769755859.923944 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b237): Mirror 3: 0xb21ae968 (stale) Mirror 4: 0xb21ae968 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b237): Mirror 3: 0xb21ae968 (stale) Mirror 4: 0xb21ae968 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b237: skipped 000112 1769755860.317492 mapwrite 0x02a17d thru 0x0316a3 (0x7527 bytes) 000114 1769755860.359630 trunc from 0x0316a4 to 0x003395 (0x2e30e bytes) 000115 1769755860.383564 mapread 0x000eb1 thru 0x003395 (0x24e5 bytes) 000116 1769755860.390874 trunc from 0x00102f to 0x003395 (0x2367 bytes) 000117 1769755860.395525 write 0x0261c3 thru 0x03350c (0xd34a bytes) 000118 1769755860.398520 write 0x03af21 thru 0x03ffff (0x50df bytes) 000119 1769755860.400444 mapread 0x0225d1 thru 0x02d859 (0xb289 bytes) 000120 1769755860.407544 mapwrite 0x01cc1a thru 0x025619 (0x8a00 bytes) 000121 1769755860.447036 mapread 0x02aed0 thru 0x02e0d2 (0x3203 bytes) 000122 1769755860.448049 mapwrite 0x00a317 thru 0x00e204 (0x3eee bytes) 000123 1769755860.491713 mapwrite 0x00f34b thru 0x010450 (0x1106 bytes) 000124 1769755860.518590 read 0x02be6f thru 0x0360a8 (0xa23a bytes) 000125 1769755860.529434 mapwrite 0x01c37b thru 0x020213 (0x3e99 bytes) 000126 1769755860.587072 trunc from 0x015507 to 0x024b85 (0xf67f bytes) 000127 1769755860.600480 fallocate 0x012e8c thru 0x01d661 (0xa7d6 bytes) 000128 1769755860.614947 fallocate 0x01a286 thru 0x01fe2c (0x5ba7 bytes) 000129 1769755860.630696 trunc from 0x00f10b to 0x01c3ab (0xd2a1 bytes) 000130 1769755860.645539 trunc from 0x040000 to 0x036554 (0x9aab bytes) 000131 1769755860.685195 punch from 0x02ad03 to 0x031427 (0x6725 bytes) 000132 1769755860.697865 mapwrite 0x0371c2 thru 0x03a131 (0x2f70 bytes) 000133 1769755860.786442 fallocate 0x024257 thru 0x02b80d (0x75b7 bytes) 000134 1769755860.795736 mapwrite 0x02cf25 thru 0x02d417 (0x4f3 bytes) 000135 1769755860.837477 read 0x039108 thru 0x03a131 (0x102a bytes) 000136 1769755860.838245 punch from 0x00db68 to 0x01c3ba (0xe853 bytes) 000137 1769755860.852033 mapread 0x0333db thru 0x035738 (0x235e bytes) 000140 1769755860.859978 mapread 0x01a334 thru 0x02475a (0xa427 bytes) 000142 1769755860.884226 mapwrite 0x02d3c0 thru 0x03b4b3 (0xe0f4 bytes) 000143 1769755860.974004 fallocate 0x017588 thru 0x01ae43 (0x38bc bytes) 000144 1769755860.986519 fallocate 0x005254 thru 0x00d361 (0x810e bytes) 000145 1769755860.999655 fallocate 0x00ff8f thru 0x0177b7 (0x7829 bytes) 000146 1769755861.021731 write 0x01fae9 thru 0x0294d6 (0x99ee bytes) 000147 1769755861.023065 trunc from 0x036d6f to 0x036ea8 (0x13a bytes) 000148 1769755861.042782 read 0x027be5 thru 0x037716 (0xfb32 bytes) 000149 1769755861.070260 trunc from 0x03b4b4 to 0x02b449 (0x1006a bytes) 000150 1769755861.114827 read 0x0073e8 thru 0x01637b (0xef94 bytes) 000151 1769755861.118454 read 0x00a096 thru 0x00d4e0 (0x344b bytes) 000152 1769755861.120689 trunc from 0x02b44a to 0x01980a (0x11c3f bytes) 000153 1769755861.149041 mapread 0x00864c thru 0x0150f5 (0xcaaa bytes) 000154 1769755861.151878 mapread 0x016079 thru 0x018647 (0x25cf bytes) 000157 1769755861.157285 trunc from 0x00fb3e to 0x01980a (0x9ccd bytes) 000158 1769755861.163930 mapread 0x00c315 thru 0x00fd38 (0x3a24 bytes) 000159 1769755861.175451 write 0x03f374 thru 0x03ffff (0xc8c bytes) 000160 1769755861.177726 read 0x019656 thru 0x01efc1 (0x596c bytes) 000161 1769755861.187434 write 0x033ab4 thru 0x03f73f (0xbc8c bytes) 000162 1769755861.195572 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x69b7abb9 (stale) Mirror 4: 0x69b7abb9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7eb7d88a Mirror 5: 0x7eb7d88a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000163 1769755862.412950 mapwrite 0x0375ff thru 0x03bf4d (0x494f bytes) 000164 1769755862.555130 write 0x01dddc thru 0x02437a (0x659f bytes) 000166 1769755862.593467 trunc from 0x040000 to 0x02edc5 (0x1123a bytes) 000167 1769755862.645889 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2edc6): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2edc6): Mirror 3: 0x453716e8 (stale) Mirror 4: 0x453716e8 (stale) Mirror 5: 0x5077b56 (stale) CRC-32 checksum value for chunk [0, 0x2edc6): Mirror 1: 0x5afc08be Mirror 6: 0x5afc08be lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2edc6: skipped 000168 1769755863.756121 trunc from 0x02edc6 to 0x01fdfa (0xefcb bytes) 000169 1769755863.858107 mapwrite 0x01fae6 thru 0x02bd91 (0xc2ac bytes) 000170 1769755863.989831 mapread 0x02016f thru 0x029fab (0x9e3d bytes) 000171 1769755863.991639 read 0x014da4 thru 0x01ff4a (0xb1a7 bytes) 000172 1769755864.025640 mapread 0x00bdd2 thru 0x012a94 (0x6cc3 bytes) 000173 1769755864.068476 trunc from 0x02bd92 to 0x013f2d (0x17e64 bytes) 000174 1769755864.091161 read 0x006eba thru 0x00ddca (0x6f11 bytes) 000175 1769755864.091890 write 0x03a62b thru 0x03b374 (0xd4a bytes) 000176 1769755864.092950 write 0x021af1 thru 0x0247bf (0x2ccf bytes) 000177 1769755864.113200 MIRROR_RESYNC 5 mirrors 000178 1769755865.784325 read 0x00d689 thru 0x016008 (0x8980 bytes) 000179 1769755865.813408 punch from 0x01dbac to 0x026632 (0x8a87 bytes) 000180 1769755865.860346 punch from 0x02a019 to 0x039ad1 (0xfab9 bytes) 000181 1769755865.870374 read 0x018788 thru 0x02464d (0xbec6 bytes) 000182 1769755865.914470 trunc from 0x001f6f to 0x005237 (0x32c9 bytes) 000183 1769755865.924188 punch from 0x016daa to 0x026b70 (0xfdc7 bytes) 000185 1769755865.930349 fallocate 0x02f842 thru 0x030496 (0xc55 bytes) 000186 1769755865.936470 write 0x002c21 thru 0x00c646 (0x9a26 bytes) 000187 1769755865.947282 read 0x0350ce thru 0x03b374 (0x62a7 bytes) 000188 1769755865.948950 trunc from 0x002c71 to 0x00dd0c (0xb09c bytes) 000189 1769755865.971906 mapread 0x02cf05 thru 0x02e46b (0x1567 bytes) 000191 1769755865.976426 write 0x0252c1 thru 0x0255bb (0x2fb bytes) 000192 1769755866.015782 mapwrite 0x026a4d thru 0x02a66b (0x3c1f bytes) 000193 1769755866.066447 mapread 0x029a65 thru 0x02c43b (0x29d7 bytes) 000194 1769755866.067985 punch from 0x00b892 to 0x00f418 (0x3b87 bytes) 000195 1769755866.075472 fallocate 0x012b55 thru 0x01b1c4 (0x8670 bytes) 000196 1769755866.083179 write 0x01b902 thru 0x021c11 (0x6310 bytes) 000197 1769755866.086295 trunc from 0x02fcd6 to 0x031a5d (0x1d88 bytes) 000198 1769755866.092755 read 0x036a4e thru 0x038e15 (0x23c8 bytes) 000199 1769755866.093754 mapread 0x032e0b thru 0x03b374 (0x856a bytes) 000200 1769755866.096379 mapwrite 0x01b527 thru 0x02778f (0xc269 bytes) 000201 1769755866.121330 fallocate 0x01365e thru 0x020ea1 (0xd844 bytes) 000202 1769755866.126086 mapwrite 0x01854d thru 0x024f48 (0xc9fc bytes) 000203 1769755866.146950 mapread 0x0353f3 thru 0x03b22c (0x5e3a bytes) 000204 1769755866.148197 write 0x00fb94 thru 0x01ea7f (0xeeec bytes) 000205 1769755866.166114 mapread 0x02a371 thru 0x0344bc (0xa14c bytes) 000206 1769755866.170823 write 0x03f58a thru 0x03ffff (0xa76 bytes) 000207 1769755866.172561 mapread 0x00d918 thru 0x01a7a9 (0xce92 bytes) 000209 1769755866.175310 write 0x011626 thru 0x018631 (0x700c bytes) 000210 1769755866.177679 read 0x0295a2 thru 0x035d67 (0xc7c6 bytes) 000211 1769755866.178518 trunc from 0x040000 to 0x017e31 (0x281ce bytes) 000212 1769755866.237552 fallocate 0x0267b9 thru 0x02d940 (0x7188 bytes) 000213 1769755866.242401 mapread 0x00b500 thru 0x00c3b0 (0xeb1 bytes) 000214 1769755866.246555 MIRROR_RESYNC 5 mirrors 000215 1769755867.548999 write 0x02971a thru 0x02d399 (0x3c80 bytes) 000216 1769755867.606962 fallocate 0x0166d7 thru 0x01a093 (0x39bd bytes) 000217 1769755867.616246 trunc from 0x02d941 to 0x027153 (0x67ed bytes) 000218 1769755867.648414 punch from 0x01ba52 to 0x01f1bd (0x376c bytes) 000219 1769755867.664780 write 0x0340d0 thru 0x03ffff (0xbf30 bytes) 000220 1769755867.666632 mapread 0x005b40 thru 0x00b26e (0x572f bytes) 000221 1769755867.699835 trunc from 0x040000 to 0x019a46 (0x265b9 bytes) 000222 1769755867.736200 trunc from 0x014830 to 0x016f0b (0x26dc bytes) 000223 1769755867.743188 read 0x010b2b thru 0x019a46 (0x8f1c bytes) 000225 1769755867.759427 mapwrite 0x0098ef thru 0x00bbb9 (0x22cb bytes) 000226 1769755867.773974 mapread 0x00167a thru 0x0063dc (0x4d63 bytes) 000227 1769755867.776125 mapwrite 0x015ac3 thru 0x01a544 (0x4a82 bytes) 000228 1769755867.842321 mapread 0x0062fb thru 0x009bda (0x38e0 bytes) 000229 1769755867.846171 trunc from 0x0172df to 0x01a544 (0x3266 bytes) 000230 1769755867.855234 trunc from 0x01a545 to 0x017460 (0x30e4 bytes) 000232 1769755867.877110 fallocate 0x023d06 thru 0x02eb1d (0xae18 bytes) 000233 1769755867.881692 trunc from 0x02b34e to 0x02eb1d (0x37d0 bytes) 000234 1769755867.891238 fallocate 0x025122 thru 0x02abef (0x5ace bytes) 000235 1769755867.895871 mapwrite 0x038a20 thru 0x03c4fc (0x3add bytes) 000236 1769755867.971127 write 0x032870 thru 0x038189 (0x591a bytes) 000237 1769755867.989427 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c4fd): Mirror 3: 0xbd40de01 (stale) Mirror 4: 0xbd40de01 (stale) Mirror 5: 0xbd40de01 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c4fd): Mirror 3: 0xbd40de01 (stale) Mirror 4: 0xbd40de01 (stale) Mirror 5: 0xbd40de01 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c4fd: skipped 000238 1769755868.534731 punch from 0x028eee to 0x029728 (0x83b bytes) 000239 1769755868.541446 punch from 0x0040b1 to 0x00afb7 (0x6f07 bytes) 000240 1769755868.548459 trunc from 0x02dfea to 0x03a9ae (0xc9c5 bytes) 000241 1769755868.560135 MIRROR_RESYNC 4 mirrors 000242 1769755870.224811 read 0x02a38e thru 0x02fd3f (0x59b2 bytes) 000243 1769755870.236489 write 0x01e7a0 thru 0x02488d (0x60ee bytes) 000245 1769755870.332901 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c4fd): Mirror 4: 0x8ff35d94 (stale) Mirror 5: 0x8ff35d94 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c4fd): Mirror 4: 0x8ff35d94 (stale) Mirror 5: 0x8ff35d94 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c4fd: skipped 000246 1769755870.837102 write 0x034d24 thru 0x03eb06 (0x9de3 bytes) 000247 1769755870.866624 mapread 0x02fe94 thru 0x031159 (0x12c6 bytes) 000248 1769755870.888831 fallocate 0x01b8bd thru 0x02ad88 (0xf4cc bytes) 000249 1769755870.905458 punch from 0x02c518 to 0x03bfcd (0xfab6 bytes) 000250 1769755870.954873 trunc from 0x03eb07 to 0x018360 (0x267a6 bytes) 000251 1769755871.014627 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x18361): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18361): Mirror 4: 0xeeb63139 (stale) Mirror 5: 0xeeb63139 (stale) CRC-32 checksum value for chunk [0, 0x18361): Mirror 1: 0xeeb63139 Mirror 6: 0xeeb63139 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18361: skipped 000252 1769755872.360546 trunc from 0x018361 to 0x007479 (0x10ee7 bytes) 000254 1769755872.418789 read 0x006393 thru 0x007479 (0x10e7 bytes) 000255 1769755872.427525 fallocate 0x02c906 thru 0x0361c9 (0x98c4 bytes) 000256 1769755872.433874 punch from 0x001da4 to 0x0054fd (0x375a bytes) 000259 1769755872.443202 punch from 0x01836e to 0x0206ee (0x8381 bytes) 000260 1769755872.449170 punch from 0x01ac36 to 0x026744 (0xbb0f bytes) 000262 1769755872.454487 mapwrite 0x00ebdb thru 0x015b7d (0x6fa3 bytes) 000263 1769755872.498122 fallocate 0x02b1d7 thru 0x033b13 (0x893d bytes) 000264 1769755872.506782 write 0x01ef41 thru 0x02b4c9 (0xc589 bytes) 000265 1769755872.528873 MIRROR_RESYNC 4 mirrors 000266 1769755873.474174 mapread 0x003033 thru 0x00ae77 (0x7e45 bytes) 000267 1769755873.509424 punch from 0x010511 to 0x01d33f (0xce2f bytes) 000268 1769755873.576835 trunc from 0x0361ca to 0x02cca6 (0x9523 bytes) 000269 1769755873.602764 mapread 0x01e5c8 thru 0x020379 (0x1db2 bytes) 000271 1769755873.612641 mapwrite 0x0116dc thru 0x01667e (0x4fa3 bytes) 000272 1769755873.635497 fallocate 0x003f83 thru 0x0100c9 (0xc147 bytes) 000273 1769755873.638912 mapwrite 0x0387da thru 0x039dad (0x15d4 bytes) 000274 1769755873.672747 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39dae): Mirror 5: 0x895845d2 (stale) Mirror 6: 0x895845d2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39dae): Mirror 5: 0x895845d2 (stale) Mirror 6: 0x895845d2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39dae: skipped 000275 1769755874.006751 trunc from 0x0332a5 to 0x039dad (0x6b09 bytes) 000276 1769755874.019457 punch from 0x0370cb to 0x039318 (0x224e bytes) 000277 1769755874.026189 MIRROR_RESYNC 3 mirrors 000278 1769755874.555660 mapread 0x005cc4 thru 0x0108cd (0xac0a bytes) 000279 1769755874.571485 read 0x0025fe thru 0x00d127 (0xab2a bytes) 000281 1769755874.574593 MIRROR_EXTEND to 4 mirrors 000282 1769755875.108751 read 0x00adfa thru 0x01747c (0xc683 bytes) 000283 1769755875.117833 write 0x01736c thru 0x017e33 (0xac8 bytes) 000284 1769755875.151660 read 0x001ac4 thru 0x006e21 (0x535e bytes) 000285 1769755875.162667 punch from 0x00ae3d to 0x00b3b3 (0x577 bytes) 000286 1769755875.167457 mapread 0x01c10c thru 0x0208ad (0x47a2 bytes) 000287 1769755875.169240 mapwrite 0x017cda thru 0x0181c6 (0x4ed bytes) 000289 1769755875.196246 read 0x00ce35 thru 0x01234d (0x5519 bytes) 000290 1769755875.197898 punch from 0x025846 to 0x026571 (0xd2c bytes) 000291 1769755875.210229 mapwrite 0x028a1a thru 0x0332e3 (0xa8ca bytes) 000292 1769755875.279119 MIRROR_RESYNC 4 mirrors 000293 1769755876.565764 mapwrite 0x027d7d thru 0x03006b (0x82ef bytes) 000294 1769755876.662366 write 0x019f46 thru 0x026e9d (0xcf58 bytes) 000295 1769755876.677823 write 0x0161c3 thru 0x01637f (0x1bd bytes) 000296 1769755876.679923 punch from 0x02043d to 0x02967a (0x923e bytes) 000297 1769755876.697085 write 0x001c7b thru 0x003f92 (0x2318 bytes) 000299 1769755876.712332 trunc from 0x039dae to 0x013d81 (0x2602c bytes) 000300 1769755876.744914 trunc from 0x013d82 to 0x014497 (0x716 bytes) 000301 1769755876.761213 mapread 0x00ccbb thru 0x014497 (0x77dd bytes) 000303 1769755876.772955 write 0x00727e thru 0x00f20b (0x7f8e bytes) 000304 1769755876.776300 read 0x004b2e thru 0x014183 (0xf656 bytes) 000305 1769755876.777360 mapread 0x00f1c9 thru 0x014497 (0x52cf bytes) 000306 1769755876.782513 trunc from 0x00d564 to 0x00e82f (0x12cc bytes) 000308 1769755876.809126 punch from 0x00f05e to 0x014497 (0x543a bytes) 000309 1769755876.815192 trunc from 0x014498 to 0x02db63 (0x196cc bytes) 000310 1769755876.833942 read 0x0175a9 thru 0x01a53a (0x2f92 bytes) 000311 1769755876.864793 punch from 0x022426 to 0x02db63 (0xb73e bytes) 000312 1769755876.870291 punch from 0x014d82 to 0x0248af (0xfb2e bytes) 000313 1769755876.876058 punch from 0x02ba81 to 0x02db63 (0x20e3 bytes) 000314 1769755876.881613 trunc from 0x01f760 to 0x02b142 (0xb9e3 bytes) 000315 1769755876.887715 read 0x02ac14 thru 0x02d0c1 (0x24ae bytes) 000316 1769755876.902491 fallocate 0x0209a0 thru 0x02fdb6 (0xf417 bytes) 000317 1769755876.908318 trunc from 0x02fdb7 to 0x00fb97 (0x2021f bytes) 000318 1769755876.939071 trunc from 0x00fb98 to 0x023148 (0x135b1 bytes) 000319 1769755876.968055 read 0x02274e thru 0x023148 (0x9fb bytes) 000320 1769755876.982764 mapwrite 0x0030c2 thru 0x00a964 (0x78a3 bytes) 000321 1769755877.004676 write 0x01090f thru 0x01fdd9 (0xf4cb bytes) 000322 1769755877.016928 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23149): Mirror 6: 0xeb53726c (stale) Mirror 7: 0xeb53726c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23149): Mirror 6: 0xeb53726c (stale) Mirror 7: 0xeb53726c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23149: skipped 000323 1769755877.335164 fallocate 0x013a4f thru 0x0202f1 (0xc8a3 bytes) 000324 1769755877.342768 trunc from 0x00141c to 0x00c10e (0xacf3 bytes) 000325 1769755877.349879 read 0x00899e thru 0x0188d1 (0xff34 bytes) 000326 1769755877.368141 trunc from 0x016a0a to 0x018642 (0x1c39 bytes) 000327 1769755877.374676 write 0x02efe1 thru 0x03a579 (0xb599 bytes) 000328 1769755877.381815 mapread 0x024f7a thru 0x0322db (0xd362 bytes) 000330 1769755877.413390 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a57a): Mirror 7: 0x333578e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a57a): Mirror 7: 0x333578e2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a57a: skipped 000331 1769755877.671092 trunc from 0x03a57a to 0x00973e (0x30e3b bytes) 000332 1769755877.693493 punch from 0x004670 to 0x00760a (0x2f9b bytes) 000333 1769755877.698591 trunc from 0x0083a9 to 0x00973e (0x1396 bytes) 000334 1769755877.702560 read 0x00292d thru 0x00407e (0x1752 bytes) 000335 1769755877.708628 fallocate 0x007f8f thru 0x00b534 (0x35a6 bytes) 000336 1769755877.713405 mapread 0x008ada thru 0x00973e (0xc65 bytes) 000337 1769755877.721892 read 0x000e67 thru 0x00296d (0x1b07 bytes) 000338 1769755877.730642 punch from 0x006b73 to 0x00973e (0x2bcc bytes) 000339 1769755877.734917 fallocate 0x02fe89 thru 0x03c0a8 (0xc220 bytes) 000340 1769755877.742845 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x973f): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x973f): Mirror 7: 0xeba5004b (stale) CRC-32 checksum value for chunk [0, 0x973f): Mirror 1: 0xeba5004b Mirror 8: 0xeba5004b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x973f: skipped 000341 1769755878.555387 write 0x000a2e thru 0x0061d9 (0x57ac bytes) 000342 1769755878.599069 mapwrite 0x004011 thru 0x012dd0 (0xedc0 bytes) 000343 1769755878.661933 fallocate 0x0098d5 thru 0x009d27 (0x453 bytes) 000344 1769755878.675323 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x12dd1): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12dd1): Mirror 7: 0xd6fc1da1 (stale) Mirror 8: 0xc369adc5 (stale) CRC-32 checksum value for chunk [0, 0x12dd1): Mirror 1: 0x7dac0e83 Mirror 9: 0x7dac0e83 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12dd1: skipped 000345 1769755879.424102 fallocate 0x00085c thru 0x00c5d4 (0xbd79 bytes) 000346 1769755879.499205 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x12dd1): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12dd1): Mirror 7: 0xd6fc1da1 (stale) Mirror 8: 0xc369adc5 (stale) Mirror 9: 0x7dac0e83 (stale) CRC-32 checksum value for chunk [0, 0x12dd1): Mirror 1: 0x7dac0e83 Mirror 10: 0x7dac0e83 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12dd1: skipped 000347 1769755880.343952 write 0x002bd7 thru 0x00b4ac (0x88d6 bytes) 000348 1769755880.390570 fallocate 0x00b476 thru 0x013849 (0x83d4 bytes) 000349 1769755880.394722 trunc from 0x01384a to 0x017190 (0x3947 bytes) 000350 1769755880.417654 MIRROR_RESYNC 5 mirrors 000351 1769755881.125244 MIRROR_RESYNC 5 mirrors 000352 1769755881.524260 fallocate 0x033c72 thru 0x03ad54 (0x70e3 bytes) 000353 1769755881.596396 write 0x01ab67 thru 0x028085 (0xd51f bytes) 000354 1769755881.614296 mapwrite 0x03da6b thru 0x03ffff (0x2595 bytes) 000355 1769755881.681540 read 0x022a1a thru 0x02350d (0xaf4 bytes) 000356 1769755881.682303 trunc from 0x03f6d5 to 0x03ffff (0x92b bytes) 000357 1769755881.691004 mapwrite 0x017a36 thru 0x01e03d (0x6608 bytes) 000358 1769755881.724392 mapwrite 0x033750 thru 0x035780 (0x2031 bytes) 000359 1769755881.761346 write 0x030805 thru 0x0385aa (0x7da6 bytes) 000361 1769755881.763303 trunc from 0x01fc11 to 0x02111a (0x150a bytes) 000362 1769755881.771230 trunc from 0x040000 to 0x00e4d8 (0x31b27 bytes) 000363 1769755881.792817 fallocate 0x02c0d8 thru 0x031273 (0x519c bytes) 000364 1769755881.799727 fallocate 0x02f598 thru 0x03b19f (0xbc08 bytes) 000365 1769755881.811368 MIRROR_RESYNC 5 mirrors 000366 1769755882.701440 read 0x033ac5 thru 0x03b19f (0x76db bytes) 000367 1769755882.708776 trunc from 0x038b64 to 0x03b19f (0x263c bytes) 000368 1769755882.731792 fallocate 0x007285 thru 0x012010 (0xad8c bytes) 000369 1769755882.736416 punch from 0x005016 to 0x00a7e8 (0x57d3 bytes) 000370 1769755882.740119 read 0x021337 thru 0x023516 (0x21e0 bytes) 000371 1769755882.749420 trunc from 0x02dc56 to 0x03965a (0xba05 bytes) 000372 1769755882.753452 trunc from 0x03b1a0 to 0x03e01e (0x2e7f bytes) 000373 1769755882.766771 trunc from 0x023c2b to 0x031800 (0xdbd6 bytes) 000374 1769755882.778223 mapwrite 0x023bf1 thru 0x02f797 (0xbba7 bytes) 000375 1769755882.849841 MIRROR_RESYNC 5 mirrors 000376 1769755884.056890 mapread 0x01b3a1 thru 0x027830 (0xc490 bytes) 000377 1769755884.082314 mapread 0x02daf4 thru 0x0307cb (0x2cd8 bytes) 000378 1769755884.090001 fallocate 0x00c379 thru 0x0128c4 (0x654c bytes) 000379 1769755884.116627 mapread 0x011e0f thru 0x015b8b (0x3d7d bytes) 000381 1769755884.129257 mapwrite 0x016bbe thru 0x02378f (0xcbd2 bytes) 000382 1769755884.159510 trunc from 0x003bf1 to 0x003e0d (0x21d bytes) 000383 1769755884.163613 trunc from 0x0121fb to 0x017545 (0x534b bytes) 000384 1769755884.168290 read 0x001832 thru 0x005c8a (0x4459 bytes) 000385 1769755884.182817 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3e01f): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e01f): Mirror 7: 0xb5af3395 (stale) Mirror 8: 0xb5af3395 (stale) Mirror 9: 0xb5af3395 (stale) Mirror 10: 0xb5af3395 (stale) CRC-32 checksum value for chunk [0, 0x3e01f): Mirror 1: 0x147d948b Mirror 11: 0x147d948b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e01f: skipped 000387 1769755884.940244 mapread 0x01728f thru 0x020bce (0x9940 bytes) 000388 1769755884.958826 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e01f): Mirror 7: 0xb5af3395 (stale) Mirror 8: 0xb5af3395 (stale) Mirror 9: 0xb5af3395 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e01f): Mirror 7: 0xb5af3395 (stale) Mirror 8: 0xb5af3395 (stale) Mirror 9: 0xb5af3395 (stale) CRC-32 checksum value for chunk [0, 0x3e01f): Mirror 1: 0x147d948b Mirror 11: 0x147d948b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e01f: skipped 000391 1769755885.914000 mapwrite 0x02559f thru 0x02b3ab (0x5e0d bytes) 000392 1769755886.073859 mapread 0x0290f2 thru 0x0368d3 (0xd7e2 bytes) 000393 1769755886.088626 fallocate 0x0261c1 thru 0x030daf (0xabef bytes) 000395 1769755886.094203 punch from 0x02b80b to 0x030bec (0x53e2 bytes) 000396 1769755886.099269 punch from 0x0131d1 to 0x0179c8 (0x47f8 bytes) 000397 1769755886.110423 mapread 0x0166e2 thru 0x026243 (0xfb62 bytes) 000398 1769755886.122388 trunc from 0x01d2bc to 0x0213ab (0x40f0 bytes) 000399 1769755886.127132 read 0x01ee9a thru 0x02cbd2 (0xdd39 bytes) 000400 1769755886.135214 trunc from 0x03e01f to 0x01fd94 (0x1e28a bytes) 000401 1769755886.173675 write 0x03bfb5 thru 0x03dd83 (0x1dcf bytes) 000402 1769755886.174864 read 0x014765 thru 0x01b337 (0x6bd3 bytes) 000403 1769755886.175548 mapwrite 0x03983e thru 0x03ffff (0x67c2 bytes) 000404 1769755886.226583 MIRROR_RESYNC 5 mirrors 000405 1769755887.540585 MIRROR_EXTEND to 6 mirrors 000406 1769755888.143697 trunc from 0x034805 to 0x03f059 (0xa855 bytes) 000407 1769755888.215134 punch from 0x019f6d to 0x024162 (0xa1f6 bytes) 000409 1769755888.228503 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x5a3fbc2b (stale) Mirror 8: 0x5a3fbc2b (stale) Mirror 9: 0x5a3fbc2b (stale) Mirror 11: 0x5a3fbc2b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x5a3fbc2b (stale) Mirror 8: 0x5a3fbc2b (stale) Mirror 9: 0x5a3fbc2b (stale) Mirror 11: 0x5a3fbc2b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000410 1769755888.632549 MIRROR_RESYNC 5 mirrors 000411 1769755889.437811 trunc from 0x040000 to 0x000e2f (0x3f1d0 bytes) 000412 1769755889.481668 fallocate 0x0333b8 thru 0x038d75 (0x59be bytes) 000413 1769755889.489756 mapread 0x034a50 thru 0x038d75 (0x4326 bytes) 000415 1769755889.521383 write 0x0280db thru 0x0366eb (0xe611 bytes) 000416 1769755889.533903 mapread 0x026e69 thru 0x031f0b (0xb0a3 bytes) 000417 1769755889.535233 trunc from 0x038d76 to 0x02522a (0x13b4b bytes) 000418 1769755889.560876 fallocate 0x00da01 thru 0x0180d4 (0xa6d4 bytes) 000419 1769755889.567832 trunc from 0x02522b to 0x02722d (0x2003 bytes) 000420 1769755889.587838 fallocate 0x007940 thru 0x01682e (0xeeef bytes) 000421 1769755889.594549 fallocate 0x017e5f thru 0x022c1d (0xadbf bytes) 000422 1769755889.607427 trunc from 0x00ac90 to 0x0176a9 (0xca1a bytes) 000423 1769755889.618991 punch from 0x01c0ef to 0x01eecb (0x2ddd bytes) 000424 1769755889.629162 trunc from 0x02722e to 0x01bb8d (0xb6a0 bytes) 000425 1769755889.651357 read 0x0184e4 thru 0x01b1c9 (0x2ce6 bytes) 000426 1769755889.660136 punch from 0x003e46 to 0x013b68 (0xfd23 bytes) 000427 1769755889.667413 punch from 0x0057eb to 0x0127da (0xcff0 bytes) 000428 1769755889.675334 mapwrite 0x014dc5 thru 0x022c88 (0xdec4 bytes) 000429 1769755889.748805 mapwrite 0x00834f thru 0x013f63 (0xbc15 bytes) 000430 1769755889.794890 punch from 0x02191c to 0x022c88 (0x136d bytes) 000431 1769755889.809236 read 0x01c965 thru 0x02274f (0x5deb bytes) 000432 1769755889.824512 mapread 0x01839b thru 0x01b134 (0x2d9a bytes) 000433 1769755889.828041 fallocate 0x03b458 thru 0x03ffff (0x4ba8 bytes) 000434 1769755889.852233 mapwrite 0x009c27 thru 0x013dac (0xa186 bytes) 000435 1769755889.876087 mapread 0x004819 thru 0x006a19 (0x2201 bytes) 000436 1769755889.883344 MIRROR_RESYNC 5 mirrors 000437 1769755890.995876 MIRROR_EXTEND to 6 mirrors 000438 1769755892.101944 mapread 0x008344 thru 0x008cf1 (0x9ae bytes) 000439 1769755892.123468 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4f:0x0]: Invalid argument (22) 000440 1769755892.436091 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4f:0x0]: Invalid argument (22) 000441 1769755892.684807 mapread 0x007622 thru 0x0148a4 (0xd283 bytes) 000442 1769755892.759209 read 0x00fd34 thru 0x01ddc7 (0xe094 bytes) 000443 1769755892.765136 trunc from 0x024e65 to 0x02ea98 (0x9c34 bytes) 000444 1769755892.818827 mapwrite 0x027f91 thru 0x02843e (0x4ae bytes) 000445 1769755892.866507 write 0x027e41 thru 0x030a35 (0x8bf5 bytes) 000446 1769755892.875520 trunc from 0x037bd4 to 0x03e3c4 (0x67f1 bytes) 000447 1769755892.882917 mapwrite 0x023a20 thru 0x024cd1 (0x12b2 bytes) 000448 1769755892.916555 mapwrite 0x001652 thru 0x009a19 (0x83c8 bytes) 000450 1769755893.020102 trunc from 0x01e13c to 0x02ca0d (0xe8d2 bytes) 000451 1769755893.029268 mapwrite 0x0142f7 thru 0x01ae24 (0x6b2e bytes) 000452 1769755893.136572 punch from 0x017c86 to 0x018ddb (0x1156 bytes) 000453 1769755893.156495 write 0x01437c thru 0x020a3b (0xc6c0 bytes) 000454 1769755893.177510 punch from 0x02429f to 0x029893 (0x55f5 bytes) 000455 1769755893.198140 read 0x0000df thru 0x00f844 (0xf766 bytes) 000456 1769755893.247483 fallocate 0x01e435 thru 0x02c0c0 (0xdc8c bytes) 000457 1769755893.252626 write 0x03dba7 thru 0x03ffff (0x2459 bytes) 000458 1769755893.254630 mapread 0x0262a4 thru 0x02680a (0x567 bytes) 000459 1769755893.255702 mapread 0x0226d8 thru 0x023647 (0xf70 bytes) 000460 1769755893.257725 write 0x0083d4 thru 0x00e4ba (0x60e7 bytes) 000461 1769755893.259817 punch from 0x012e3f to 0x01aeff (0x80c1 bytes) 000462 1769755893.327529 mapread 0x01b869 thru 0x028af3 (0xd28b bytes) 000463 1769755893.345431 fallocate 0x005e2c thru 0x00b819 (0x59ee bytes) 000464 1769755893.354477 punch from 0x0055b6 to 0x00ceac (0x78f7 bytes) 000465 1769755893.362072 trunc from 0x040000 to 0x005705 (0x3a8fa bytes) 000466 1769755893.428306 punch from 0x00465c to 0x005705 (0x10aa bytes) 000467 1769755893.437709 mapwrite 0x026590 thru 0x02cd91 (0x6802 bytes) 000468 1769755893.621199 trunc from 0x02cd92 to 0x00d198 (0x1fbf9 bytes) 000469 1769755893.654556 punch from 0x00aaa6 to 0x00d02a (0x2585 bytes) 000470 1769755893.664565 mapread 0x00816e thru 0x00d198 (0x502b bytes) 000472 1769755893.710395 punch from 0x001233 to 0x00cc22 (0xb9f0 bytes) 000473 1769755893.718309 fallocate 0x02e7dc thru 0x033401 (0x4c26 bytes) 000474 1769755893.735573 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33402): Mirror 8: 0x88f0a4f (stale) Mirror 12: 0x88f0a4f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33402): Mirror 8: 0x88f0a4f (stale) Mirror 12: 0x88f0a4f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33402: skipped 000475 1769755894.152213 fallocate 0x030d6a thru 0x0345d0 (0x3867 bytes) 000476 1769755894.162552 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x345d1): Mirror 8: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x345d1): Mirror 8: 0x2bbff09f (stale) Mirror 12: 0x2bbff09f (stale) CRC-32 checksum value for chunk [0, 0x345d1): Mirror 1: 0x9a1876c9 Mirror 13: 0x9a1876c9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x345d1: skipped 000478 1769755895.339455 mapread 0x0002f4 thru 0x00dbfd (0xd90a bytes) 000479 1769755895.388870 read 0x026d85 thru 0x0345d0 (0xd84c bytes) 000480 1769755895.409342 write 0x01bcb9 thru 0x01e21f (0x2567 bytes) 000481 1769755895.503823 read 0x02c0db thru 0x030cf0 (0x4c16 bytes) 000483 1769755895.533247 trunc from 0x0345d1 to 0x028ebf (0xb711 bytes) 000484 1769755895.575526 read 0x00e778 thru 0x017bdf (0x9468 bytes) 000486 1769755895.577277 fallocate 0x0139a5 thru 0x0144df (0xb3b bytes) 000487 1769755895.590799 read 0x01647b thru 0x0176b3 (0x1239 bytes) 000488 1769755895.592428 fallocate 0x0133e1 thru 0x01e442 (0xb062 bytes) 000489 1769755895.616551 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x28ec0): Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28ec0): Mirror 8: 0xfafc3c5e (stale) Mirror 12: 0xfafc3c5e (stale) Mirror 13: 0xc6f4acbc (stale) CRC-32 checksum value for chunk [0, 0x28ec0): Mirror 1: 0x66d1c7aa Mirror 14: 0x66d1c7aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28ec0: skipped 000490 1769755897.071786 MIRROR_RESYNC 5 mirrors 000491 1769755897.809944 write 0x018a0a thru 0x01ab50 (0x2147 bytes) 000492 1769755897.843228 MIRROR_RESYNC 5 mirrors 000493 1769755898.577502 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4f:0x0]: Invalid argument (22) 000495 1769755898.970168 fallocate 0x016b71 thru 0x01bd22 (0x51b2 bytes) 000496 1769755899.028668 mapread 0x023ff3 thru 0x028ebf (0x4ecd bytes) 000497 1769755899.051194 MIRROR_RESYNC 4 mirrors 000499 1769755899.814169 mapwrite 0x038a6b thru 0x03ffff (0x7595 bytes) 000500 1769755899.892500 fallocate 0x0049e6 thru 0x011724 (0xcd3f bytes) 000501 1769755899.901458 MIRROR_RESYNC 4 mirrors 000503 1769755900.700515 write 0x01b2f6 thru 0x023ed2 (0x8bdd bytes) 000504 1769755900.773378 mapread 0x010f22 thru 0x011f51 (0x1030 bytes) 000505 1769755900.784668 punch from 0x028846 to 0x02bf10 (0x36cb bytes) 000506 1769755900.790005 trunc from 0x040000 to 0x01a0c8 (0x25f37 bytes) 000507 1769755900.818509 mapread 0x005f2d thru 0x007ed0 (0x1fa4 bytes) 000508 1769755900.836605 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a0c9): Mirror 12: 0x9a8c3dd5 (stale) Mirror 13: 0x9a8c3dd5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a0c9): Mirror 12: 0x9a8c3dd5 (stale) Mirror 13: 0x9a8c3dd5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a0c9: skipped 000509 1769755901.147668 trunc from 0x00edc8 to 0x019ccd (0xaf06 bytes) 000510 1769755901.155632 punch from 0x00c440 to 0x01a0c8 (0xdc89 bytes) 000511 1769755901.163341 read 0x00c6b7 thru 0x00fede (0x3828 bytes) 000512 1769755901.173757 mapwrite 0x00e8c9 thru 0x01d821 (0xef59 bytes) 000513 1769755901.293178 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1d822): Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d822): Mirror 12: 0xe1720a60 (stale) Mirror 13: 0xe1720a60 (stale) CRC-32 checksum value for chunk [0, 0x1d822): Mirror 1: 0xf1224640 Mirror 14: 0xf1224640 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d822: skipped 000515 1769755902.120125 trunc from 0x012548 to 0x018eec (0x69a5 bytes) 000516 1769755902.155592 punch from 0x00c16d to 0x0141e2 (0x8076 bytes) 000517 1769755902.160855 fallocate 0x03ff54 thru 0x03ffff (0xac bytes) 000518 1769755902.167257 trunc from 0x01d822 to 0x03e38e (0x20b6d bytes) 000519 1769755902.226570 read 0x006a2d thru 0x00adca (0x439e bytes) 000520 1769755902.235862 mapwrite 0x0379b6 thru 0x03df35 (0x6580 bytes) 000521 1769755902.279727 mapwrite 0x00753e thru 0x015088 (0xdb4b bytes) 000522 1769755902.309892 write 0x00248e thru 0x010fe5 (0xeb58 bytes) 000523 1769755902.314330 write 0x00ad36 thru 0x018b0b (0xddd6 bytes) 000524 1769755902.318282 MIRROR_RESYNC 4 mirrors 000525 1769755903.029529 trunc from 0x03e38f to 0x01f88e (0x1eb00 bytes) 000526 1769755903.093997 trunc from 0x01f88f to 0x0389a7 (0x19119 bytes) 000527 1769755903.114685 trunc from 0x031635 to 0x0389a7 (0x7373 bytes) 000528 1769755903.123156 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x389a8): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x389a8): Mirror 12: 0xc3eaab8d (stale) Mirror 13: 0xc3eaab8d (stale) Mirror 14: 0xc3eaab8d (stale) CRC-32 checksum value for chunk [0, 0x389a8): Mirror 1: 0xe2f75948 Mirror 15: 0xe2f75948 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x389a8: skipped 000531 1769755903.991485 mapwrite 0x02f540 thru 0x03c4cd (0xcf8e bytes) 000534 1769755904.072710 mapread 0x00be1f thru 0x01886b (0xca4d bytes) 000535 1769755904.094673 trunc from 0x037fb1 to 0x03c34d (0x439d bytes) 000536 1769755904.099386 mapread 0x035be5 thru 0x0368e1 (0xcfd bytes) 000537 1769755904.100411 trunc from 0x03c4ce to 0x024f02 (0x175cb bytes) 000538 1769755904.125996 trunc from 0x01727a to 0x02450d (0xd294 bytes) 000539 1769755904.139797 write 0x038570 thru 0x03ffff (0x7a90 bytes) 000540 1769755904.142226 fallocate 0x0337b3 thru 0x03ffff (0xc84d bytes) 000541 1769755904.150228 trunc from 0x03addd to 0x03ffff (0x5223 bytes) 000542 1769755904.164308 write 0x03e171 thru 0x03ffff (0x1e8f bytes) 000543 1769755904.178936 mapwrite 0x01933a thru 0x0269c7 (0xd68e bytes) 000544 1769755904.238705 write 0x027ba0 thru 0x03462e (0xca8f bytes) 000545 1769755904.246160 fallocate 0x02507f thru 0x026659 (0x15db bytes) 000546 1769755904.253697 read 0x013999 thru 0x020678 (0xcce0 bytes) 000547 1769755904.261658 mapread 0x016d2e thru 0x01a6b6 (0x3989 bytes) 000548 1769755904.265656 mapread 0x03b264 thru 0x03ffff (0x4d9c bytes) 000549 1769755904.267909 read 0x02312b thru 0x02408e (0xf64 bytes) 000550 1769755904.269531 write 0x0275fd thru 0x0293d8 (0x1ddc bytes) 000551 1769755904.270550 mapwrite 0x01b28b thru 0x024947 (0x96bd bytes) 000552 1769755904.301658 trunc from 0x017e5c to 0x01d1d8 (0x537d bytes) 000553 1769755904.308684 trunc from 0x040000 to 0x032b45 (0xd4ba bytes) 000554 1769755904.339084 trunc from 0x032b46 to 0x03b710 (0x8bcb bytes) 000555 1769755904.361638 trunc from 0x00354f to 0x00b099 (0x7b4b bytes) 000556 1769755904.370118 trunc from 0x02c225 to 0x03638b (0xa167 bytes) 000557 1769755904.378905 trunc from 0x03b711 to 0x016624 (0x250ec bytes) 000558 1769755904.405703 mapread 0x00ca75 thru 0x016624 (0x9bb0 bytes) 000559 1769755904.407172 read 0x00d027 thru 0x0100b3 (0x308d bytes) 000560 1769755904.408880 write 0x02af1b thru 0x02cc8e (0x1d74 bytes) 000561 1769755904.411682 trunc from 0x021923 to 0x022acb (0x11a9 bytes) 000562 1769755904.422556 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2cc8f): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cc8f): Mirror 12: 0xf7dbab7c (stale) Mirror 13: 0xf7dbab7c (stale) Mirror 14: 0xf7dbab7c (stale) Mirror 15: 0xf7dbab7c (stale) CRC-32 checksum value for chunk [0, 0x2cc8f): Mirror 1: 0xf2302da0 Mirror 16: 0xf2302da0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cc8f: skipped 000563 1769755905.316161 fallocate 0x03c2ce thru 0x03f43e (0x3171 bytes) 000564 1769755905.352443 read 0x02f990 thru 0x03945c (0x9acd bytes) 000565 1769755905.358693 punch from 0x015263 to 0x01e569 (0x9307 bytes) 000566 1769755905.363685 read 0x028a97 thru 0x02ce29 (0x4393 bytes) 000567 1769755905.377155 read 0x03c39b thru 0x03f43e (0x30a4 bytes) 000568 1769755905.378796 punch from 0x0310f0 to 0x03688a (0x579b bytes) 000569 1769755905.382723 trunc from 0x02c149 to 0x02e092 (0x1f4a bytes) 000570 1769755905.392100 write 0x034b57 thru 0x03ffff (0xb4a9 bytes) 000571 1769755905.406846 read 0x009806 thru 0x0185a0 (0xed9b bytes) 000572 1769755905.407455 write 0x0146f5 thru 0x01d186 (0x8a92 bytes) 000573 1769755905.410347 mapwrite 0x039950 thru 0x03ffff (0x66b0 bytes) 000574 1769755905.436565 mapread 0x01982d thru 0x01ba2e (0x2202 bytes) 000575 1769755905.441052 read 0x01254a thru 0x01fde2 (0xd899 bytes) 000576 1769755905.443270 trunc from 0x01bc7e to 0x024d91 (0x9114 bytes) 000577 1769755905.449195 mapread 0x01fd27 thru 0x02b724 (0xb9fe bytes) 000578 1769755905.466058 write 0x0187cd thru 0x024044 (0xb878 bytes) 000581 1769755905.472623 write 0x01113b thru 0x01e5ac (0xd472 bytes) 000582 1769755905.475086 fallocate 0x03a079 thru 0x03d6c3 (0x364b bytes) 000583 1769755905.483674 mapread 0x00c699 thru 0x01adaa (0xe712 bytes) 000584 1769755905.490214 fallocate 0x01a66e thru 0x02a307 (0xfc9a bytes) 000585 1769755905.498431 write 0x0050b9 thru 0x006ffb (0x1f43 bytes) 000587 1769755905.505700 trunc from 0x040000 to 0x0082f9 (0x37d06 bytes) 000588 1769755905.548266 read 0x004edd thru 0x0082f9 (0x341d bytes) 000589 1769755905.548901 trunc from 0x0017af to 0x0082f9 (0x6b4b bytes) 000590 1769755905.563560 mapread 0x0017ba thru 0x007108 (0x594f bytes) 000591 1769755905.582653 write 0x0128aa thru 0x014d62 (0x24b9 bytes) 000593 1769755905.584525 punch from 0x00b55d to 0x014d62 (0x9806 bytes) 000594 1769755905.605594 trunc from 0x0085d7 to 0x014d62 (0xc78c bytes) 000596 1769755905.619714 MIRROR_RESYNC 6 mirrors 000597 1769755906.649926 punch from 0x00db0d to 0x0129ff (0x4ef3 bytes) 000598 1769755906.697191 write 0x020916 thru 0x0256f0 (0x4ddb bytes) 000599 1769755906.698631 read 0x005080 thru 0x00edd4 (0x9d55 bytes) 000600 1769755906.710850 mapwrite 0x036c4e thru 0x03ab26 (0x3ed9 bytes) 000601 1769755906.765231 punch from 0x01458e to 0x017eaa (0x391d bytes) 000602 1769755906.773841 mapwrite 0x01b05e thru 0x020742 (0x56e5 bytes) 000603 1769755906.829163 read 0x009646 thru 0x00c9e6 (0x33a1 bytes) 000605 1769755906.830583 write 0x020f59 thru 0x029738 (0x87e0 bytes) 000606 1769755906.854708 trunc from 0x03ab27 to 0x01035c (0x2a7ca bytes) 000607 1769755906.885195 mapread 0x0016a2 thru 0x00e256 (0xcbb5 bytes) 000608 1769755906.886616 trunc from 0x0072c0 to 0x008901 (0x1642 bytes) 000609 1769755906.907075 punch from 0x003892 to 0x00cbee (0x935d bytes) 000610 1769755906.918273 mapread 0x00707e thru 0x008ba9 (0x1b2c bytes) 000611 1769755906.927970 trunc from 0x00b30c to 0x01035c (0x5051 bytes) 000612 1769755906.938617 punch from 0x0048a4 to 0x01035c (0xbab9 bytes) 000613 1769755906.945989 punch from 0x00bb83 to 0x01035c (0x47da bytes) 000615 1769755906.951205 read 0x00e6c5 thru 0x01035c (0x1c98 bytes) 000617 1769755906.972157 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1035d): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1035d): Mirror 12: 0x8fd25b17 (stale) Mirror 13: 0x8fd25b17 (stale) Mirror 14: 0x8fd25b17 (stale) Mirror 15: 0x8fd25b17 (stale) Mirror 16: 0x8fd25b17 (stale) CRC-32 checksum value for chunk [0, 0x1035d): Mirror 1: 0x8fd25b17 Mirror 17: 0x8fd25b17 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1035d: skipped 000618 1769755907.641414 punch from 0x00591e to 0x0067ff (0xee2 bytes) 000619 1769755907.678702 punch from 0x00eae8 to 0x01035c (0x1875 bytes) 000620 1769755907.686204 mapread 0x0062b9 thru 0x01035c (0xa0a4 bytes) 000621 1769755907.707551 mapread 0x005791 thru 0x00c85a (0x70ca bytes) 000622 1769755907.708811 mapwrite 0x01ed08 thru 0x02ead0 (0xfdc9 bytes) 000623 1769755907.776411 fallocate 0x033424 thru 0x03e79a (0xb377 bytes) 000624 1769755907.784865 trunc from 0x03e79b to 0x03550e (0x928c bytes) 000625 1769755907.812429 write 0x01fd30 thru 0x02a3f8 (0xa6c9 bytes) 000626 1769755907.816084 mapread 0x007b30 thru 0x00c14c (0x461d bytes) 000627 1769755907.818877 mapwrite 0x01fc7b thru 0x02c86a (0xcbf0 bytes) 000628 1769755907.840754 punch from 0x02942f to 0x03253a (0x910c bytes) 000629 1769755907.847454 trunc from 0x025da3 to 0x0261e4 (0x442 bytes) 000630 1769755907.852327 read 0x006d42 thru 0x00a45d (0x371c bytes) 000631 1769755907.854301 trunc from 0x00f3bb to 0x00fe76 (0xabc bytes) 000632 1769755907.859646 write 0x031dbd thru 0x0371fe (0x5442 bytes) 000633 1769755907.876048 trunc from 0x0371ff to 0x039573 (0x2375 bytes) 000634 1769755907.890162 write 0x03aac9 thru 0x03ffff (0x5537 bytes) 000635 1769755907.892562 write 0x02aa0a thru 0x032af9 (0x80f0 bytes) 000637 1769755907.895252 punch from 0x00de2e to 0x0131bb (0x538e bytes) 000638 1769755907.904594 MIRROR_RESYNC 7 mirrors 000639 1769755909.122317 trunc from 0x040000 to 0x039933 (0x66cc bytes) 000640 1769755909.158507 mapread 0x00fd7f thru 0x010f42 (0x11c4 bytes) 000641 1769755909.170237 read 0x028f04 thru 0x032dfd (0x9efa bytes) 000642 1769755909.178975 trunc from 0x03360f to 0x039636 (0x6028 bytes) 000643 1769755909.184205 mapwrite 0x01f0b5 thru 0x02334f (0x429b bytes) 000644 1769755909.207142 punch from 0x0060cc to 0x0114a4 (0xb3d9 bytes) 000645 1769755909.211596 punch from 0x02404d to 0x033953 (0xf907 bytes) 000646 1769755909.216831 write 0x011a80 thru 0x01d782 (0xbd03 bytes) 000647 1769755909.228199 mapread 0x012b70 thru 0x016b70 (0x4001 bytes) 000648 1769755909.229509 trunc from 0x00ac1e to 0x00ef9f (0x4382 bytes) 000649 1769755909.236401 read 0x00869d thru 0x01507b (0xc9df bytes) 000650 1769755909.246229 trunc from 0x039934 to 0x02d917 (0xc01c bytes) 000651 1769755909.263166 write 0x035046 thru 0x03ffff (0xafba bytes) 000652 1769755909.265057 fallocate 0x03b72c thru 0x03ffff (0x48d4 bytes) 000653 1769755909.270521 read 0x015343 thru 0x019c6d (0x492b bytes) 000654 1769755909.271654 mapread 0x03a9e3 thru 0x03d9f5 (0x3013 bytes) 000656 1769755909.273428 trunc from 0x02cf7f to 0x03102c (0x40ae bytes) 000657 1769755909.278471 fallocate 0x01c2c5 thru 0x021bf7 (0x5933 bytes) 000658 1769755909.284930 read 0x018f26 thru 0x027464 (0xe53f bytes) 000659 1769755909.286179 write 0x014d3e thru 0x01a522 (0x57e5 bytes) 000660 1769755909.288081 read 0x02c11e thru 0x02d8ee (0x17d1 bytes) 000661 1769755909.296183 trunc from 0x040000 to 0x03c382 (0x3c7d bytes) 000662 1769755909.310610 read 0x02f46e thru 0x035b9e (0x6731 bytes) 000664 1769755909.312853 mapwrite 0x005f72 thru 0x00eac1 (0x8b50 bytes) 000665 1769755909.337187 fallocate 0x00fbda thru 0x01c015 (0xc43c bytes) 000666 1769755909.346151 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c383): Mirror 12: 0x30469825 (stale) Mirror 13: 0x30469825 (stale) Mirror 15: 0x30469825 (stale) Mirror 16: 0x30469825 (stale) Mirror 17: 0x30469825 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c383): Mirror 12: 0x30469825 (stale) Mirror 13: 0x30469825 (stale) Mirror 15: 0x30469825 (stale) Mirror 16: 0x30469825 (stale) Mirror 17: 0x30469825 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c383: skipped 000667 1769755909.721447 MIRROR_RESYNC 6 mirrors 000668 1769755911.330924 trunc from 0x014679 to 0x01c78d (0x8115 bytes) 000669 1769755911.404697 fallocate 0x033d41 thru 0x03860f (0x48cf bytes) 000670 1769755911.412208 fallocate 0x027bf3 thru 0x02ed13 (0x7121 bytes) 000671 1769755911.423080 punch from 0x00417f to 0x004433 (0x2b5 bytes) 000672 1769755911.445825 read 0x013908 thru 0x02366d (0xfd66 bytes) 000673 1769755911.456347 write 0x00008e thru 0x00d217 (0xd18a bytes) 000674 1769755911.468492 trunc from 0x03c383 to 0x012fb0 (0x293d2 bytes) 000675 1769755911.501082 mapwrite 0x03a1fe thru 0x03ffff (0x5e02 bytes) 000676 1769755911.562293 trunc from 0x040000 to 0x037456 (0x8ba9 bytes) 000677 1769755911.582422 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37457): Mirror 12: 0x542f616e (stale) Mirror 15: 0x542f616e (stale) Mirror 16: 0x542f616e (stale) Mirror 17: 0x542f616e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37457): Mirror 12: 0x542f616e (stale) Mirror 15: 0x542f616e (stale) Mirror 16: 0x542f616e (stale) Mirror 17: 0x542f616e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37457: skipped 000678 1769755912.063881 mapread 0x01824d thru 0x018d40 (0xaf4 bytes) 000679 1769755912.070121 read 0x0207e2 thru 0x02a944 (0xa163 bytes) 000680 1769755912.089614 mapread 0x02413b thru 0x02f993 (0xb859 bytes) 000681 1769755912.104269 trunc from 0x027515 to 0x02b15a (0x3c46 bytes) 000682 1769755912.113624 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x37457): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37457): Mirror 12: 0x542f616e (stale) Mirror 15: 0x542f616e (stale) Mirror 16: 0x542f616e (stale) Mirror 17: 0x542f616e (stale) CRC-32 checksum value for chunk [0, 0x37457): Mirror 1: 0x55a11bd4 Mirror 18: 0x55a11bd4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37457: skipped 000683 1769755912.827117 read 0x001441 thru 0x010dfc (0xf9bc bytes) 000684 1769755912.836294 mapread 0x0308d4 thru 0x037456 (0x6b83 bytes) 000685 1769755912.852209 fallocate 0x0148ac thru 0x01d015 (0x876a bytes) 000686 1769755912.891179 mapread 0x0018da thru 0x00c8b9 (0xafe0 bytes) 000689 1769755912.902812 mapwrite 0x02373c thru 0x02fea9 (0xc76e bytes) 000690 1769755912.932846 trunc from 0x00297f to 0x003483 (0xb05 bytes) 000691 1769755912.936733 mapread 0x02de0f thru 0x037456 (0x9648 bytes) 000693 1769755912.944124 punch from 0x00d55a to 0x017125 (0x9bcc bytes) 000694 1769755912.948604 write 0x018d4b thru 0x024546 (0xb7fc bytes) 000696 1769755912.963900 mapread 0x004e6e thru 0x006987 (0x1b1a bytes) 000697 1769755912.969621 MIRROR_RESYNC 6 mirrors 000698 1769755913.933480 punch from 0x026d6e to 0x02fe6e (0x9101 bytes) 000699 1769755913.964514 mapread 0x0305f0 thru 0x031ad4 (0x14e5 bytes) 000700 1769755913.975311 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37457): Mirror 12: 0x1df17396 (stale) Mirror 16: 0x1df17396 (stale) Mirror 17: 0x1df17396 (stale) Mirror 18: 0x1df17396 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37457): Mirror 12: 0x1df17396 (stale) Mirror 16: 0x1df17396 (stale) Mirror 17: 0x1df17396 (stale) Mirror 18: 0x1df17396 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37457: skipped 000701 1769755914.166375 read 0x00fa02 thru 0x012b8d (0x318c bytes) 000702 1769755914.169728 mapread 0x0030c4 thru 0x007f51 (0x4e8e bytes) 000703 1769755914.182382 mapwrite 0x02ed77 thru 0x034375 (0x55ff bytes) 000704 1769755914.199681 punch from 0x01d5ec to 0x023dba (0x67cf bytes) 000705 1769755914.202760 mapread 0x025a1f thru 0x027ed6 (0x24b8 bytes) 000706 1769755914.207959 fallocate 0x02c1d2 thru 0x03936b (0xd19a bytes) 000707 1769755914.211578 trunc from 0x017fae to 0x02230a (0xa35d bytes) 000708 1769755914.214771 read 0x01e5e2 thru 0x02c2b3 (0xdcd2 bytes) 000709 1769755914.221396 write 0x005133 thru 0x00e8aa (0x9778 bytes) 000711 1769755914.225557 mapread 0x036304 thru 0x03936b (0x3068 bytes) 000712 1769755914.226734 fallocate 0x00c0d9 thru 0x00d0cd (0xff5 bytes) 000713 1769755914.230135 read 0x028dad thru 0x0324a5 (0x96f9 bytes) 000714 1769755914.230711 mapwrite 0x01bdbc thru 0x0291c5 (0xd40a bytes) 000715 1769755914.246372 read 0x038cf9 thru 0x03936b (0x673 bytes) 000716 1769755914.247091 mapwrite 0x002269 thru 0x005b30 (0x38c8 bytes) 000717 1769755914.259003 trunc from 0x00b34b to 0x01a442 (0xf0f8 bytes) 000718 1769755914.265388 write 0x001f26 thru 0x00235c (0x437 bytes) 000719 1769755914.266954 punch from 0x030e7d to 0x03399d (0x2b21 bytes) 000720 1769755914.270932 read 0x005272 thru 0x00e4c5 (0x9254 bytes) 000721 1769755914.279358 trunc from 0x03936c to 0x03b0e5 (0x1d7a bytes) 000722 1769755914.296561 write 0x03be18 thru 0x03ffff (0x41e8 bytes) 000723 1769755914.303392 punch from 0x03a4b9 to 0x03ffff (0x5b47 bytes) 000724 1769755914.322068 read 0x030628 thru 0x0366f7 (0x60d0 bytes) 000725 1769755914.322798 fallocate 0x025f6f thru 0x034cff (0xed91 bytes) 000726 1769755914.329417 trunc from 0x040000 to 0x0222e0 (0x1dd1f bytes) 000727 1769755914.339552 trunc from 0x022233 to 0x0222e0 (0xae bytes) 000729 1769755914.343887 trunc from 0x0222e1 to 0x013a04 (0xe8dc bytes) 000730 1769755914.358767 fallocate 0x01bd73 thru 0x024c8a (0x8f18 bytes) 000731 1769755914.362766 fallocate 0x004175 thru 0x0074c0 (0x334c bytes) 000733 1769755914.366398 write 0x03694f thru 0x03853a (0x1bec bytes) 000734 1769755914.367431 mapread 0x00b8cf thru 0x014a1d (0x914f bytes) 000735 1769755914.375746 fallocate 0x00743f thru 0x012065 (0xac27 bytes) 000736 1769755914.379515 punch from 0x00cc94 to 0x010f55 (0x42c2 bytes) 000737 1769755914.383673 trunc from 0x03853b to 0x000b6c (0x379ce bytes) 000738 1769755914.402535 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xb6d): Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb6d): Mirror 12: 0x39e3e7f9 (stale) Mirror 16: 0x39e3e7f9 (stale) Mirror 17: 0x39e3e7f9 (stale) Mirror 18: 0x39e3e7f9 (stale) CRC-32 checksum value for chunk [0, 0xb6d): Mirror 1: 0x39e3e7f9 Mirror 19: 0x39e3e7f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb6d: skipped 000740 1769755914.940886 mapread 0x0005d5 thru 0x000b6c (0x598 bytes) 000741 1769755914.947136 trunc from 0x000b6d to 0x02854f (0x279e3 bytes) 000742 1769755915.030360 mapread 0x003cef thru 0x012655 (0xe967 bytes) 000743 1769755915.060616 read 0x0260c1 thru 0x02854f (0x248f bytes) 000744 1769755915.065823 trunc from 0x01903d to 0x0232ac (0xa270 bytes) 000745 1769755915.070446 mapwrite 0x00335c thru 0x011772 (0xe417 bytes) 000746 1769755915.089314 trunc from 0x026780 to 0x02854f (0x1dd0 bytes) 000747 1769755915.098346 MIRROR_RESYNC 6 mirrors 000748 1769755915.989611 read 0x0201e9 thru 0x02854f (0x8367 bytes) 000749 1769755915.995355 trunc from 0x00389e to 0x00a416 (0x6b79 bytes) 000750 1769755916.020224 read 0x0008f0 thru 0x00d221 (0xc932 bytes) 000751 1769755916.027529 trunc from 0x01ea5a to 0x023027 (0x45ce bytes) 000753 1769755916.033784 mapread 0x0174cf thru 0x026002 (0xeb34 bytes) 000755 1769755916.062404 write 0x0361a8 thru 0x03fea5 (0x9cfe bytes) 000756 1769755916.065959 mapwrite 0x019ccc thru 0x01be6f (0x21a4 bytes) 000757 1769755916.085825 trunc from 0x03fea6 to 0x018a06 (0x2749f bytes) 000758 1769755916.138056 MIRROR_RESYNC 6 mirrors 000761 1769755917.249670 MIRROR_EXTEND to 7 mirrors 000762 1769755917.741528 write 0x02d469 thru 0x03cc29 (0xf7c1 bytes) 000763 1769755917.777631 trunc from 0x03cc2a to 0x019c70 (0x22fb9 bytes) 000764 1769755917.794919 trunc from 0x009d3c to 0x019445 (0xf70a bytes) 000765 1769755917.803004 write 0x02bc5a thru 0x03b5ee (0xf995 bytes) 000766 1769755917.804870 trunc from 0x020737 to 0x02d715 (0xcfdf bytes) 000767 1769755917.817961 mapwrite 0x03e316 thru 0x03ffff (0x1cea bytes) 000768 1769755917.863765 punch from 0x03b3d3 to 0x03ffff (0x4c2d bytes) 000769 1769755917.871071 mapwrite 0x010795 thru 0x01a93c (0xa1a8 bytes) 000772 1769755917.959001 punch from 0x029f4d to 0x036206 (0xc2ba bytes) 000773 1769755917.979580 punch from 0x0330d6 to 0x039e1a (0x6d45 bytes) 000774 1769755917.998815 read 0x006126 thru 0x00ad90 (0x4c6b bytes) 000775 1769755918.011466 mapread 0x01aaa4 thru 0x0282f6 (0xd853 bytes) 000776 1769755918.033613 mapread 0x00f72b thru 0x01a40e (0xace4 bytes) 000777 1769755918.040378 trunc from 0x01ee16 to 0x02a726 (0xb911 bytes) 000778 1769755918.055376 write 0x02bddd thru 0x02febc (0x40e0 bytes) 000779 1769755918.066832 fallocate 0x008c3f thru 0x010bd1 (0x7f93 bytes) 000780 1769755918.071820 mapwrite 0x03fd17 thru 0x03ffff (0x2e9 bytes) 000782 1769755918.092889 punch from 0x022c91 to 0x02cab9 (0x9e29 bytes) 000783 1769755918.100238 punch from 0x03e64b to 0x03ee8c (0x842 bytes) 000784 1769755918.104576 mapread 0x000549 thru 0x00c252 (0xbd0a bytes) 000785 1769755918.112730 trunc from 0x01375a to 0x019fae (0x6855 bytes) 000786 1769755918.116957 read 0x00b0c6 thru 0x00cac6 (0x1a01 bytes) 000787 1769755918.117729 mapread 0x02f1d1 thru 0x03f056 (0xfe86 bytes) 000788 1769755918.126739 punch from 0x026432 to 0x034362 (0xdf31 bytes) 000789 1769755918.130397 mapwrite 0x02ceee thru 0x033944 (0x6a57 bytes) 000791 1769755918.176092 trunc from 0x040000 to 0x00c9fb (0x33604 bytes) 000792 1769755918.191545 fallocate 0x014eb5 thru 0x017682 (0x27ce bytes) 000793 1769755918.196341 trunc from 0x00c9fc to 0x03efee (0x325f3 bytes) 000794 1769755918.206082 mapwrite 0x01a86b thru 0x0250c3 (0xa859 bytes) 000795 1769755918.280575 trunc from 0x03efef to 0x020a7f (0x1e56f bytes) 000796 1769755918.299138 trunc from 0x020a80 to 0x011c41 (0xee3e bytes) 000797 1769755918.319637 trunc from 0x00c120 to 0x00ec37 (0x2b18 bytes) 000798 1769755918.324974 fallocate 0x029da4 thru 0x0327c6 (0x8a23 bytes) 000799 1769755918.333147 MIRROR_RESYNC 7 mirrors 000800 1769755919.921766 write 0x0195ee thru 0x01c086 (0x2a99 bytes) 000801 1769755920.004750 trunc from 0x007bd9 to 0x014013 (0xc43b bytes) 000802 1769755920.011040 write 0x02e21b thru 0x03138b (0x3171 bytes) 000803 1769755920.012308 punch from 0x0218ba to 0x02efa4 (0xd6eb bytes) 000804 1769755920.062598 punch from 0x01aff6 to 0x02521d (0xa228 bytes) 000805 1769755920.070986 write 0x01e8e6 thru 0x02d9a7 (0xf0c2 bytes) 000806 1769755920.086203 write 0x0064ad thru 0x00c281 (0x5dd5 bytes) 000807 1769755920.103945 fallocate 0x01f749 thru 0x02b77a (0xc032 bytes) 000808 1769755920.130743 read 0x01959a thru 0x023498 (0x9eff bytes) 000811 1769755920.144102 write 0x03e573 thru 0x03ffff (0x1a8d bytes) 000812 1769755920.150116 fallocate 0x025b28 thru 0x0283fc (0x28d5 bytes) 000813 1769755920.158934 mapwrite 0x014f55 thru 0x01bc0c (0x6cb8 bytes) 000814 1769755920.229005 trunc from 0x040000 to 0x001314 (0x3eceb bytes) 000816 1769755920.260626 mapread 0x000c7e thru 0x001314 (0x697 bytes) 000817 1769755920.273231 punch from 0x000e6f to 0x001314 (0x4a6 bytes) 000818 1769755920.278292 trunc from 0x000a4e to 0x001314 (0x8c7 bytes) 000819 1769755920.287372 fallocate 0x01f2c8 thru 0x025657 (0x6390 bytes) 000820 1769755920.293851 trunc from 0x025658 to 0x0167aa (0xeead bytes) 000821 1769755920.316952 read 0x003f74 thru 0x005ea5 (0x1f32 bytes) 000822 1769755920.333387 trunc from 0x0167ab to 0x00188d (0x14f1d bytes) 000823 1769755920.351050 read 0x000d53 thru 0x00188d (0xb3b bytes) 000824 1769755920.357356 MIRROR_RESYNC 7 mirrors 000825 1769755921.413683 trunc from 0x00188e to 0x0242b5 (0x22a28 bytes) 000826 1769755921.488270 mapread 0x01ece6 thru 0x0242b5 (0x55d0 bytes) 000827 1769755921.506966 punch from 0x00de17 to 0x013934 (0x5b1e bytes) 000828 1769755921.511823 read 0x01a342 thru 0x02357c (0x923b bytes) 000829 1769755921.516836 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x242b6): Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x242b6): Mirror 12: 0x2daec3ac (stale) Mirror 16: 0x2daec3ac (stale) Mirror 17: 0x2daec3ac (stale) Mirror 18: 0x2daec3ac (stale) Mirror 19: 0x2daec3ac (stale) Mirror 20: 0x2daec3ac (stale) CRC-32 checksum value for chunk [0, 0x242b6): Mirror 1: 0x2daec3ac Mirror 21: 0x2daec3ac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x242b6: skipped 000830 1769755922.254616 fallocate 0x019aa1 thru 0x028402 (0xe962 bytes) 000831 1769755922.302587 write 0x011310 thru 0x018bb6 (0x78a7 bytes) 000832 1769755922.339381 mapread 0x01ea33 thru 0x0232ec (0x48ba bytes) 000833 1769755922.352961 read 0x01c9ff thru 0x028402 (0xba04 bytes) 000834 1769755922.359545 punch from 0x017018 to 0x0234da (0xc4c3 bytes) 000835 1769755922.388864 read 0x013f4e thru 0x0174aa (0x355d bytes) 000836 1769755922.401186 read 0x00545f thru 0x00f444 (0x9fe6 bytes) 000837 1769755922.401919 trunc from 0x00935a to 0x00d825 (0x44cc bytes) 000838 1769755922.409577 read 0x01cf70 thru 0x026aaf (0x9b40 bytes) 000839 1769755922.420787 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x28403): Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28403): Mirror 12: 0x67eb6042 (stale) Mirror 16: 0x67eb6042 (stale) Mirror 17: 0x67eb6042 (stale) Mirror 18: 0x67eb6042 (stale) Mirror 19: 0x67eb6042 (stale) Mirror 20: 0x67eb6042 (stale) Mirror 21: 0x67eb6042 (stale) CRC-32 checksum value for chunk [0, 0x28403): Mirror 1: 0x7ffef1d1 Mirror 22: 0x7ffef1d1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28403: skipped 000840 1769755923.289823 read 0x01620c thru 0x01a07d (0x3e72 bytes) 000843 1769755923.326742 write 0x0347e6 thru 0x039c62 (0x547d bytes) 000844 1769755923.390179 punch from 0x0397b1 to 0x039c62 (0x4b2 bytes) 000845 1769755923.405397 write 0x034249 thru 0x034de3 (0xb9b bytes) 000846 1769755923.407862 punch from 0x008c6d to 0x010b05 (0x7e99 bytes) 000848 1769755923.419093 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x39c63): Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39c63): Mirror 12: 0x817d1931 (stale) Mirror 16: 0x817d1931 (stale) Mirror 17: 0x817d1931 (stale) Mirror 18: 0x817d1931 (stale) Mirror 19: 0x817d1931 (stale) Mirror 20: 0x817d1931 (stale) Mirror 21: 0x817d1931 (stale) Mirror 22: 0x4f64a8da (stale) CRC-32 checksum value for chunk [0, 0x39c63): Mirror 1: 0x6733cca5 Mirror 23: 0x6733cca5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39c63: skipped 000849 1769755924.358269 punch from 0x00b1df to 0x00b71f (0x541 bytes) 000850 1769755924.395956 trunc from 0x039c63 to 0x028077 (0x11beb bytes) 000852 1769755924.422522 fallocate 0x00a6a9 thru 0x00c800 (0x2158 bytes) 000853 1769755924.427278 fallocate 0x01d5e1 thru 0x0279a9 (0xa3c9 bytes) 000854 1769755924.433479 mapread 0x01a131 thru 0x025bf8 (0xbac8 bytes) 000855 1769755924.462132 trunc from 0x028078 to 0x03a64c (0x125d5 bytes) 000856 1769755924.477583 mapwrite 0x0282c3 thru 0x035e6e (0xdbac bytes) 000857 1769755924.553859 trunc from 0x03a64d to 0x00c73c (0x2df10 bytes) 000858 1769755924.585566 fallocate 0x01c779 thru 0x02a1cb (0xda53 bytes) 000859 1769755924.608817 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc73d): Mirror 12: 0xfa24a21e (stale) Mirror 16: 0xfa24a21e (stale) Mirror 17: 0xfa24a21e (stale) Mirror 18: 0xfa24a21e (stale) Mirror 19: 0xfa24a21e (stale) Mirror 21: 0xfa24a21e (stale) Mirror 22: 0xfa24a21e (stale) Mirror 23: 0xfa24a21e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc73d): Mirror 12: 0xfa24a21e (stale) Mirror 16: 0xfa24a21e (stale) Mirror 17: 0xfa24a21e (stale) Mirror 18: 0xfa24a21e (stale) Mirror 19: 0xfa24a21e (stale) Mirror 21: 0xfa24a21e (stale) Mirror 22: 0xfa24a21e (stale) Mirror 23: 0xfa24a21e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc73d: skipped 000860 1769755925.028123 fallocate 0x00bb98 thru 0x01b980 (0xfde9 bytes) 000862 1769755925.033211 mapread 0x001b2c thru 0x0023ae (0x883 bytes) 000863 1769755925.044725 punch from 0x00251a to 0x006270 (0x3d57 bytes) 000864 1769755925.049053 read 0x003e24 thru 0x00c73c (0x8919 bytes) 000865 1769755925.057076 write 0x0214a4 thru 0x030c22 (0xf77f bytes) 000868 1769755925.063886 read 0x00b18e thru 0x0165bd (0xb430 bytes) 000870 1769755925.073339 trunc from 0x030c23 to 0x01dd95 (0x12e8d bytes) 000871 1769755925.085379 punch from 0x00e385 to 0x0161ed (0x7e69 bytes) 000872 1769755925.094167 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1dd96): Mirror 12: 0xbd091edb (stale) Mirror 16: 0xbd091edb (stale) Mirror 17: 0xbd091edb (stale) Mirror 18: 0xbd091edb (stale) Mirror 19: 0xbd091edb (stale) Mirror 21: 0xbd091edb (stale) Mirror 22: 0xb33ba9da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dd96): Mirror 12: 0xbd091edb (stale) Mirror 16: 0xbd091edb (stale) Mirror 17: 0xbd091edb (stale) Mirror 18: 0xbd091edb (stale) Mirror 19: 0xbd091edb (stale) Mirror 21: 0xbd091edb (stale) Mirror 22: 0xb33ba9da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dd96: skipped 000874 1769755925.524819 write 0x00adbe thru 0x00cb5c (0x1d9f bytes) 000875 1769755925.542886 mapread 0x012fa5 thru 0x01dd95 (0xadf1 bytes) 000876 1769755925.559658 trunc from 0x00ba94 to 0x01402f (0x859c bytes) 000877 1769755925.566943 fallocate 0x004a56 thru 0x0105e3 (0xbb8e bytes) 000879 1769755925.574560 MIRROR_RESYNC 8 mirrors 000880 1769755926.656799 trunc from 0x01dd96 to 0x01c6db (0x16ba bytes) 000881 1769755926.690483 read 0x009a73 thru 0x00cbed (0x317b bytes) 000882 1769755926.695840 punch from 0x01b792 to 0x01c6db (0xf4a bytes) 000884 1769755926.699544 fallocate 0x00eae7 thru 0x0105df (0x1af9 bytes) 000885 1769755926.702869 fallocate 0x0289f7 thru 0x030d7c (0x8386 bytes) 000886 1769755926.714121 MIRROR_RESYNC 8 mirrors 000887 1769755927.924851 trunc from 0x01c6dc to 0x0366b8 (0x19fdd bytes) 000888 1769755927.979776 mapread 0x00d7fd thru 0x01c033 (0xe837 bytes) 000889 1769755928.009975 read 0x027747 thru 0x03049e (0x8d58 bytes) 000890 1769755928.030448 read 0x012ad0 thru 0x014faf (0x24e0 bytes) 000892 1769755928.031698 read 0x01746b thru 0x0243b7 (0xcf4d bytes) 000893 1769755928.050165 read 0x00e849 thru 0x0122ed (0x3aa5 bytes) 000895 1769755928.051492 punch from 0x011a4e to 0x013912 (0x1ec5 bytes) 000896 1769755928.059803 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x366b9): Mirror 12: 0x805e6c78 (stale) Mirror 16: 0x805e6c78 (stale) Mirror 17: 0x805e6c78 (stale) Mirror 18: 0x805e6c78 (stale) Mirror 21: 0x805e6c78 (stale) Mirror 22: 0x805e6c78 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x366b9): Mirror 12: 0x805e6c78 (stale) Mirror 16: 0x805e6c78 (stale) Mirror 17: 0x805e6c78 (stale) Mirror 18: 0x805e6c78 (stale) Mirror 21: 0x805e6c78 (stale) Mirror 22: 0x805e6c78 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x366b9: skipped 000897 1769755928.639338 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x366b9): Mirror 16: 0x805e6c78 (stale) Mirror 17: 0x805e6c78 (stale) Mirror 18: 0x805e6c78 (stale) Mirror 21: 0x805e6c78 (stale) Mirror 22: 0x805e6c78 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x366b9): Mirror 16: 0x805e6c78 (stale) Mirror 17: 0x805e6c78 (stale) Mirror 18: 0x805e6c78 (stale) Mirror 21: 0x805e6c78 (stale) Mirror 22: 0x805e6c78 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x366b9: skipped 000898 1769755929.107458 fallocate 0x02cc8c thru 0x033b3f (0x6eb4 bytes) 000899 1769755929.113503 read 0x022ebf thru 0x029f60 (0x70a2 bytes) 000900 1769755929.125189 mapwrite 0x002cb1 thru 0x007af0 (0x4e40 bytes) 000901 1769755929.159334 read 0x02f9f0 thru 0x0366b8 (0x6cc9 bytes) 000902 1769755929.175385 mapwrite 0x02866e thru 0x035e16 (0xd7a9 bytes) 000903 1769755929.203951 fallocate 0x036e7c thru 0x03ffff (0x9184 bytes) 000905 1769755929.215400 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x366b9): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x366b9): Mirror 16: 0x805e6c78 (stale) Mirror 17: 0x805e6c78 (stale) Mirror 18: 0x805e6c78 (stale) Mirror 21: 0x805e6c78 (stale) Mirror 22: 0x805e6c78 (stale) CRC-32 checksum value for chunk [0, 0x366b9): Mirror 1: 0xf9fecdca Mirror 23: 0xf9fecdca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x366b9: skipped 000906 1769755930.213042 read 0x025be7 thru 0x0321e8 (0xc602 bytes) 000907 1769755930.250979 MIRROR_RESYNC 7 mirrors 000908 1769755932.054456 fallocate 0x020e1d thru 0x02f8d0 (0xeab4 bytes) 000909 1769755932.094805 read 0x017b82 thru 0x01eacb (0x6f4a bytes) 000910 1769755932.104473 trunc from 0x0366b9 to 0x02e370 (0x8348 bytes) 000911 1769755932.155282 read 0x02b78f thru 0x02e370 (0x2be2 bytes) 000912 1769755932.166317 fallocate 0x00de51 thru 0x012797 (0x4947 bytes) 000913 1769755932.171311 fallocate 0x0007e0 thru 0x004448 (0x3c69 bytes) 000914 1769755932.176266 trunc from 0x02e371 to 0x01fdb5 (0xe5bb bytes) 000915 1769755932.198281 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1fdb6): Mirror 16: 0xd2e5909a (stale) Mirror 18: 0xd2e5909a (stale) Mirror 21: 0xd2e5909a (stale) Mirror 22: 0xd2e5909a (stale) Mirror 23: 0xd2e5909a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fdb6): Mirror 16: 0xd2e5909a (stale) Mirror 18: 0xd2e5909a (stale) Mirror 21: 0xd2e5909a (stale) Mirror 22: 0xd2e5909a (stale) Mirror 23: 0xd2e5909a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fdb6: skipped 000916 1769755932.543503 MIRROR_RESYNC 6 mirrors 000917 1769755933.429980 trunc from 0x00e573 to 0x017cdb (0x9769 bytes) 000918 1769755933.465778 mapread 0x016700 thru 0x01fdb5 (0x96b6 bytes) 000919 1769755933.484272 fallocate 0x00f9ed thru 0x01d94d (0xdf61 bytes) 000920 1769755933.489757 punch from 0x01414a to 0x01fdb5 (0xbc6c bytes) 000921 1769755933.494061 fallocate 0x027c48 thru 0x02e2df (0x6698 bytes) 000922 1769755933.497977 write 0x00c470 thru 0x013e3b (0x79cc bytes) 000923 1769755933.509669 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2e2e0): Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e2e0): Mirror 16: 0xe7aa464f (stale) Mirror 18: 0xe7aa464f (stale) Mirror 21: 0xe7aa464f (stale) Mirror 22: 0xe7aa464f (stale) Mirror 23: 0xe7aa464f (stale) CRC-32 checksum value for chunk [0, 0x2e2e0): Mirror 1: 0xd61ab9ec Mirror 24: 0xd61ab9ec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e2e0: skipped 000924 1769755934.237910 trunc from 0x00e8c0 to 0x01c40b (0xdb4c bytes) 000926 1769755934.284378 punch from 0x028577 to 0x02909b (0xb25 bytes) 000927 1769755934.288938 trunc from 0x02e2e0 to 0x01f2b1 (0xf02e bytes) 000928 1769755934.338287 punch from 0x019c1e to 0x01a079 (0x45c bytes) 000929 1769755934.346894 MIRROR_RESYNC 7 mirrors 000930 1769755935.429841 punch from 0x00ce0c to 0x01742d (0xa622 bytes) 000932 1769755935.455674 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1f2b2): Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f2b2): Mirror 16: 0x74b6c6af (stale) Mirror 18: 0x74b6c6af (stale) Mirror 21: 0x74b6c6af (stale) Mirror 22: 0x74b6c6af (stale) Mirror 23: 0x74b6c6af (stale) Mirror 24: 0x74b6c6af (stale) CRC-32 checksum value for chunk [0, 0x1f2b2): Mirror 1: 0xe743bd73 Mirror 25: 0xe743bd73 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f2b2: skipped 000934 1769755936.467679 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1f2b2): Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f2b2): Mirror 16: 0x74b6c6af (stale) Mirror 18: 0x74b6c6af (stale) Mirror 21: 0x74b6c6af (stale) Mirror 22: 0x74b6c6af (stale) Mirror 23: 0x74b6c6af (stale) Mirror 24: 0x74b6c6af (stale) CRC-32 checksum value for chunk [0, 0x1f2b2): Mirror 1: 0xe743bd73 Mirror 25: 0xe743bd73 Mirror 26: 0xe743bd73 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f2b2: skipped 000935 1769755937.539613 fallocate 0x00507c thru 0x007234 (0x21b9 bytes) 000936 1769755937.576713 trunc from 0x01f2b2 to 0x013a96 (0xb81b bytes) 000937 1769755937.601541 punch from 0x007222 to 0x007b69 (0x948 bytes) 000938 1769755937.607076 mapread 0x00b971 thru 0x013a96 (0x8126 bytes) 000940 1769755937.627915 mapwrite 0x0149e0 thru 0x01f76a (0xad8b bytes) 000941 1769755937.680782 trunc from 0x00456b to 0x00d0e4 (0x8b7a bytes) 000942 1769755937.688688 punch from 0x0001ed to 0x000cc8 (0xadc bytes) 000943 1769755937.694352 fallocate 0x030231 thru 0x035b94 (0x5964 bytes) 000944 1769755937.702844 punch from 0x004041 to 0x013055 (0xf015 bytes) 000945 1769755937.719220 mapread 0x01dbd8 thru 0x01f76a (0x1b93 bytes) 000946 1769755937.722898 trunc from 0x01f76b to 0x036759 (0x16fef bytes) 000947 1769755937.753449 punch from 0x028e0b to 0x033f83 (0xb179 bytes) 000948 1769755937.758245 trunc from 0x00d5be to 0x0107d4 (0x3217 bytes) 000949 1769755937.773701 trunc from 0x03675a to 0x01478c (0x21fcd bytes) 000950 1769755937.804059 trunc from 0x01478d to 0x0078e6 (0xcea6 bytes) 000951 1769755937.826124 read 0x002a7b thru 0x002d69 (0x2ef bytes) 000952 1769755937.844794 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x78e7): Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x78e7): Mirror 16: 0x4948ab9e (stale) Mirror 18: 0x4948ab9e (stale) Mirror 21: 0x4948ab9e (stale) Mirror 22: 0x4948ab9e (stale) Mirror 23: 0x4948ab9e (stale) Mirror 24: 0x4948ab9e (stale) Mirror 25: 0x4948ab9e (stale) Mirror 26: 0x4948ab9e (stale) CRC-32 checksum value for chunk [0, 0x78e7): Mirror 1: 0x48b98b64 Mirror 27: 0x48b98b64 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x78e7: skipped 000953 1769755938.981686 punch from 0x000a48 to 0x0078e6 (0x6e9f bytes) 000954 1769755939.021154 trunc from 0x0056db to 0x0078e6 (0x220c bytes) 000955 1769755939.025778 punch from 0x005ce9 to 0x0078e6 (0x1bfe bytes) 000956 1769755939.029987 punch from 0x004c45 to 0x0078e6 (0x2ca2 bytes) 000957 1769755939.034625 write 0x037c53 thru 0x03ffff (0x83ad bytes) 000958 1769755939.042772 trunc from 0x040000 to 0x01a1af (0x25e50 bytes) 000959 1769755939.069131 write 0x01ebec thru 0x0297e3 (0xabf8 bytes) 000960 1769755939.071893 mapread 0x002724 thru 0x0055b8 (0x2e95 bytes) 000961 1769755939.087345 punch from 0x01cc8d to 0x01e471 (0x17e5 bytes) 000962 1769755939.103069 punch from 0x014979 to 0x01e077 (0x96ff bytes) 000964 1769755939.109483 mapread 0x018ed3 thru 0x019906 (0xa34 bytes) 000965 1769755939.125525 mapwrite 0x017fb3 thru 0x01f1cf (0x721d bytes) 000966 1769755939.144900 mapwrite 0x03046c thru 0x03c51e (0xc0b3 bytes) 000967 1769755939.203768 read 0x006607 thru 0x00ddca (0x77c4 bytes) 000968 1769755939.204617 mapread 0x017c6a thru 0x020649 (0x89e0 bytes) 000969 1769755939.207686 mapread 0x034379 thru 0x038e73 (0x4afb bytes) 000970 1769755939.209357 read 0x0205ca thru 0x02d260 (0xcc97 bytes) 000971 1769755939.210426 mapread 0x0227ee thru 0x028a7e (0x6291 bytes) 000972 1769755939.211975 write 0x02c3fc thru 0x036806 (0xa40b bytes) 000973 1769755939.215203 write 0x003a23 thru 0x01102e (0xd60c bytes) 000974 1769755939.217610 write 0x0106c4 thru 0x01dda9 (0xd6e6 bytes) 000975 1769755939.219995 read 0x003fb1 thru 0x012737 (0xe787 bytes) 000976 1769755939.221360 write 0x038451 thru 0x03ffff (0x7baf bytes) 000977 1769755939.223481 trunc from 0x040000 to 0x036581 (0x9a7e bytes) 000978 1769755939.237548 fallocate 0x03220c thru 0x03ffff (0xddf4 bytes) 000979 1769755939.242870 write 0x02267a thru 0x02dc48 (0xb5cf bytes) 000980 1769755939.244313 mapwrite 0x035af7 thru 0x038212 (0x271c bytes) 000981 1769755939.289940 mapread 0x030d8a thru 0x033c41 (0x2eb8 bytes) 000982 1769755939.293316 trunc from 0x038213 to 0x00dafb (0x2a717 bytes) 000983 1769755939.352362 mapread 0x004dfe thru 0x00d062 (0x8265 bytes) 000984 1769755939.353574 mapwrite 0x03f528 thru 0x03ffff (0xad8 bytes) 000985 1769755939.433949 punch from 0x0110d1 to 0x01898d (0x78bd bytes) 000986 1769755939.446973 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xb163d7f4 (stale) Mirror 18: 0xb163d7f4 (stale) Mirror 21: 0xb163d7f4 (stale) Mirror 22: 0xb163d7f4 (stale) Mirror 23: 0xb163d7f4 (stale) Mirror 24: 0xb163d7f4 (stale) Mirror 25: 0x78b68451 (stale) Mirror 26: 0x78b68451 (stale) Mirror 27: 0xd6abbfe4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x92e19c1e Mirror 28: 0x92e19c1e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000987 1769755940.259187 trunc from 0x03734d to 0x03ea5d (0x7711 bytes) 000988 1769755940.305751 punch from 0x02fbd4 to 0x03a56f (0xa99c bytes) 000991 1769755940.316571 fallocate 0x021355 thru 0x0297c1 (0x846d bytes) 000992 1769755940.322831 punch from 0x03794b to 0x03e196 (0x684c bytes) 000993 1769755940.331461 punch from 0x00bfe4 to 0x016454 (0xa471 bytes) 000994 1769755940.336215 mapwrite 0x01e9d6 thru 0x021673 (0x2c9e bytes) 000995 1769755940.364277 read 0x030f15 thru 0x037c4c (0x6d38 bytes) 000996 1769755940.372336 punch from 0x015f3b to 0x023f9b (0xe061 bytes) 000997 1769755940.376930 fallocate 0x03594b thru 0x03721f (0x18d5 bytes) 000998 1769755940.381732 write 0x038668 thru 0x03eb99 (0x6532 bytes) 000999 1769755940.393339 mapwrite 0x02b83c thru 0x03869d (0xce62 bytes) 001000 1769755940.419680 read 0x012132 thru 0x017b7f (0x5a4e bytes) All operations completed A-OK! PASS 70a (96s) == sanity-flr test 71: check mirror extend parent fid ==== 01:52:25 (1769755945) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x88:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (9s) == sanity-flr test 72: check append on FLR file ========== 01:52:35 (1769755955) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.230986 s, 9.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098164 s, 10.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb9:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 01:52:42 (1769755962) Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre2 Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 8255104 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 989504 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1664832 .. Extending file size to 8682560 .. Extending file size to 9134048 .. Extending file size to 8685792 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8899520 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 59948 59949 59951 59953 59955 192.168.203.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg350-client.virtnet /mnt/lustre2 (opts:) 192.168.203.150@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg350-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (76s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 01:53:58 (1769756038) Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre2 Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7353824 .. Extending file size to 6179552 .. Extending file size to 8494432 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7928544 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8397344 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8845536 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8720512 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8904032 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9309824 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61868 61869 61872 61873 61874 failed 192.168.203.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg350-client.virtnet /mnt/lustre2 (opts:) 192.168.203.150@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg350-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (76s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 01:55:14 (1769756114) Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2492 1268812 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2120 1269184 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24996 3582024 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 19056 3587964 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 44052 7169988 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2492 1268812 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2120 1269184 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24996 3582024 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 19056 3587964 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 44052 7169988 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0731402 s, 28.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.238822 s, 8.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xb9520a73 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg350-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 01:55:32 (1769756132) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0690526 s, 30.4 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 01:55:38 (1769756138) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x97:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xce:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 01:55:45 (1769756145) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0808097 s, 26.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x28e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x28e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x28f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 01:55:51 (1769756151) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.108856 s, 19.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x28f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x290:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x290:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x28f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x290:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x290:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x291:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 01:55:58 (1769756158) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0762784 s, 27.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x291:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x292:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x291:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x292:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x293:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 01:56:30 (1769756190) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg350-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697858 s, 15.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x292:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x295:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x294:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x292:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x295:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x294:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x293:0x0] } PASS 204d (48s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 01:57:19 (1769756239) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg350-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0936374 s, 22.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (52s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 01:58:10 (1769756290) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg350-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg350-server: Pool lustre.test_204f created Adding targets to pool oleg350-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg350-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.133273 s, 15.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg350-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg350-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg350-server: Pool lustre.test_204f destroyed PASS 204f (71s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 01:59:21 (1769756361) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 01:59:29 (1769756369) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 01:59:39 (1769756379) oleg350-server: Pool lustre.test_206 created Adding targets to pool oleg350-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg350-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg350-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg350-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg350-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 02:00:07 (1769756407) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107536 s, 9.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206207 s, 50.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xd5c4ab60 Mirror 3: 0xd5c4ab60 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 02:00:17 (1769756417) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 02:00:21 (1769756421) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 02:00:24 (1769756424) Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0472406 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x7b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1730000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1730000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1730000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xdf0000 Read this (4 bytes): BBBB 192.168.203.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg350-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 02:01:01 (1769756461) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0261684 s, 313 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 02:01:07 (1769756467) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503846 s, 20.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb4 lmm_fid: [0x200000405:0xb4:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb4 lmm_fid: [0x200000405:0xb4:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xdc:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611316 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-30 02:01:08.000000000 -0500 Modify: 2026-01-30 02:01:08.000000000 -0500 Change: 2026-01-30 02:01:10.000000000 -0500 Birth: 2026-01-30 02:01:09.000000000 -0500 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 02:01:15 (1769756475) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539039 s, 19.5 MB/s oleg350-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg350-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg350-server: O/280000401/d28/dc: File not found by ext2_lookup PASS 210b (24s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 02:01:39 (1769756499) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.182295 s, 225 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0462653 s, 88.5 kB/s size after second write -rw-r--r-- 1 root root 4096 Jan 30 02:01 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xb9:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jan 30 02:01 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 02:01:48 (1769756508) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 02:01:54 (1769756514) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1696 sec =========== 02:02:02 (1769756522) === sanity-flr: start cleanup 02:02:04 (1769756524) === === sanity-flr: finish cleanup 02:02:13 (1769756533) ===