-----============= acceptance-small: sanity-flr ============----- Mon Nov 17 12:50:55 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg452-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 12:51:13 (1763401873) === oleg452-client.virtnet: executing check_config_client /mnt/lustre oleg452-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg452-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff94d9081fc800.idle_timeout=debug osc.lustre-OST0001-osc-ffff94d9081fc800.idle_timeout=debug disable quota as required oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 12:51:42 (1763401902) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7562] == sanity-flr test 0a: lfs mirror create with -N option == 12:51:45 (1763401905) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:52:01 (1763401921) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:52:04 (1763401924) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 12:52:07 (1763401927) PASS 0d (21s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:52:28 (1763401948) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:52:31 (1763401951) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 12:52:33 (1763401953) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff94d9081fc800.stats=clear osc.lustre-OST0001-osc-ffff94d9081fc800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:52:40 (1763401960) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (8s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:52:48 (1763401968) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:52:55 (1763401975) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 12:52:58 (1763401978) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (7s) == sanity-flr test 2: create components from existing files ========================================================== 12:53:05 (1763401985) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:53:11 (1763401991) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:53:14 (1763401994) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:53:20 (1763402000) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (5s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 12:53:29 (1763402009) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:53:35 (1763402015) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.330042 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12024 s, 8.7 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:53:43 (1763402023) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.127724 s, 41.0 MB/s PASS 22 (6s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:53:49 (1763402029) Stopping /mnt/lustre-ost1 (opts:) on oleg452-server oleg452-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94d9081fc800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94d9081fc800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg452-server: oleg452-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg452-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg452-server oleg452-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff94d9081fc800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94d9081fc800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg452-server: oleg452-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg452-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (52s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:54:41 (1763402081) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.322311 s, 29.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg452-server oleg452-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94d9081fc800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94d9081fc800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg452-server: oleg452-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg452-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (35s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:55:16 (1763402116) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg452-server oleg452-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94d9081fc800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94d9081fc800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg452-server: oleg452-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg452-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg452-server oleg452-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff94d9081fc800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94d9081fc800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg452-server: oleg452-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg452-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (59s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:56:16 (1763402176) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.95629 s, 13.3 MB/s -rw-r--r-- 1 root root 52428800 Nov 17 12:56 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.982s user 0m0.001s sys 0m0.190s Stopping /mnt/lustre-ost1 (opts:) on oleg452-server oleg452-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94d9081fc800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94d9081fc800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.532s user 0m0.001s sys 0m0.194s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg452-server: oleg452-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg452-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg452-server oleg452-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff94d9081fc800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94d9081fc800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.291s user 0m0.002s sys 0m0.181s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg452-server: oleg452-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg452-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg452-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff94d9081fc800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94d9081fc800.ost_server_uuid in FULL state after 0 sec PASS 33b (77s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:57:32 (1763402252) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:57:35 (1763402255) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:57:38 (1763402258) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 12:57:42 (1763402262) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 12:57:49 (1763402269) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.545338 s, 23.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (10s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:58:00 (1763402280) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213906 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356632 s, 29.4 MB/s PASS 36b (5s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:58:05 (1763402285) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:58:13 (1763402293) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00078903 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000686276 s, 1.5 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00151897 s, 674 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00105334 s, 972 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 12:58:22 (1763402302) create test files with size 15756798 .. a04969c27a4d1555396346ab007ace12 - 330fa591c1ef482b64d6430086f7cffc - f3bb545a5d11df6afeed7e21f54077ce - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg452-client.virtnet /mnt/lustre (opts:) Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 12:58:44 (1763402324) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:58:57 (1763402337) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:59:02 (1763402342) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121998 s, 8.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191389 s, 54.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 12:59:10 (1763402350) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131795 s, 15.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.407146 s, 10.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.203607 s, 10.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (12s) == sanity-flr test 42: lfs mirror verify ================= 12:59:22 (1763402362) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 12:59:25 (1763402365) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 12:59:28 (1763402368) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171588 s, 61.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 12:59:33 (1763402373) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 12:59:36 (1763402376) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4e:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (6s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:59:44 (1763402384) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.224348 s, 46.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg452-client.virtnet /mnt/lustre (opts:) Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:59:53 (1763402393) llite.lustre-ffff94d9081fe000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.221673 s, 47.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg452-client.virtnet /mnt/lustre (opts:) Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff94d903a0f800.xattr_cache=1 PASS 44e (11s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 13:00:04 (1763402404) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (29s) == sanity-flr test 46: Verify setstripe --copy option ==== 13:00:33 (1763402433) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (32s) == sanity-flr test 47: Verify mirror obj alloc =========== 13:01:05 (1763402465) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 13:01:07 (1763402467) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.257795 s, 12.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.14989 s, 21.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xa2bc4032 (stale) Mirror 2: 0xa2bc4032 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 80f14b05a74995f617d9259a6145c6d4 - mirror 1 checksum: e675c8d150a3531c1e7d34ca65770101 - mirror 2 checksum: 80f14b05a74995f617d9259a6145c6d4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Nov 17 13:01:09 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Nov 17 13:01:09 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 80f14b05a74995f617d9259a6145c6d4 - mirror 1 checksum: e675c8d150a3531c1e7d34ca65770101 - mirror 2 checksum: e675c8d150a3531c1e7d34ca65770101 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Nov 17 13:01:09 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Nov 17 13:01:09 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } PASS 48 (9s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 13:01:17 (1763402477) Failing mds1 on oleg452-server Stopping /mnt/lustre-mds1 (opts:) on oleg452-server 13:01:25 (1763402485) shut down facet: mds1 facet_host: oleg452-server facet_failover_host: oleg452-server Failover mds1 to oleg452-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-MDT0000 13:01:55 (1763402515) targets are mounted 13:01:55 (1763402515) facet_failover done oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660675 s, 15.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 50A (50s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 13:02:07 (1763402527) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 13:02:10 (1763402530) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 13:02:13 (1763402533) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0067749 s, 2.4 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 13:02:20 (1763402540) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 13:02:23 (1763402543) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 13:02:26 (1763402546) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (19s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 13:02:45 (1763402565) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 13:03:12 (1763402592) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 13:03:35 (1763402615) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (22s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 13:03:58 (1763402638) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (5s) == sanity-flr test 70: mirror create and split race ====== 13:04:04 (1763402644) mirror create pid 52292 mirror split pid 52293 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 52292 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 52293 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (65s) == sanity-flr test 70a: flr mode fsx test ================ 13:05:09 (1763402709) Seed set to 2711 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1763402711.774886 mapwrite 0x00cf97 thru 0x0139de (0x6a48 bytes) truncating to largest ever: 0xd204 000002 1763402711.924984 trunc from 0x0139df to 0x00d203 (0x67db bytes) truncating to largest ever: 0x3bac3 000003 1763402711.951235 trunc from 0x00d204 to 0x03bac2 (0x2e8bf bytes) 000004 1763402711.981407 write 0x0331d8 thru 0x038c52 (0x5a7b bytes) 000006 1763402711.998908 MIRROR_EXTEND to 2 mirrors 000009 1763402712.743521 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) 000010 1763402712.974009 mapwrite 0x006cd3 thru 0x008be6 (0x1f14 bytes) 000012 1763402713.090015 MIRROR_EXTEND to 2 mirrors 000013 1763402713.928147 MIRROR_RESYNC 2 mirrors 000014 1763402714.239179 MIRROR_RESYNC 2 mirrors 000017 1763402714.687763 mapread 0x02ae30 thru 0x0353fd (0xa5ce bytes) 000019 1763402714.740631 MIRROR_EXTEND to 3 mirrors 000021 1763402715.573726 mapwrite 0x010537 thru 0x010a3a (0x504 bytes) truncating to largest ever: 0x3e815 000024 1763402715.757537 trunc from 0x03bac3 to 0x03e814 (0x2d52 bytes) 000025 1763402715.792546 MIRROR_RESYNC 3 mirrors 000026 1763402716.577017 read 0x00bc77 thru 0x0167b6 (0xab40 bytes) 000027 1763402716.590419 read 0x012103 thru 0x01e787 (0xc685 bytes) 000030 1763402716.604877 mapwrite 0x004156 thru 0x00a282 (0x612d bytes) 000031 1763402716.791849 mapread 0x039bdb thru 0x03e814 (0x4c3a bytes) 000034 1763402716.845372 write 0x016e27 thru 0x0241af (0xd389 bytes) 000036 1763402716.867782 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e815): Mirror 3: 0x6223a82c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e815): Mirror 3: 0x6223a82c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e815: skipped 000038 1763402717.177512 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e815): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e815): Mirror 3: 0x6223a82c (stale) CRC-32 checksum value for chunk [0, 0x3e815): Mirror 1: 0x316a0e5a Mirror 4: 0x316a0e5a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e815: skipped 000044 1763402718.017462 mapread 0x021bf0 thru 0x0235d7 (0x19e8 bytes) 000046 1763402718.067845 read 0x03198c thru 0x037ec7 (0x653c bytes) 000048 1763402718.095016 trunc from 0x03e815 to 0x0381c5 (0x664f bytes) 000049 1763402718.184529 mapread 0x016d6d thru 0x0184c6 (0x175a bytes) 000051 1763402718.203271 mapwrite 0x01bc9a thru 0x0205a2 (0x4909 bytes) 000052 1763402718.287827 read 0x01a509 thru 0x01e860 (0x4358 bytes) 000053 1763402718.289029 mapread 0x01088b thru 0x01bd30 (0xb4a6 bytes) 000055 1763402718.290400 read 0x007e57 thru 0x01408e (0xc238 bytes) 000057 1763402718.301897 read 0x00cab9 thru 0x018a00 (0xbf48 bytes) 000058 1763402718.303344 mapread 0x006e22 thru 0x0108a9 (0x9a88 bytes) 000059 1763402718.304212 mapread 0x02bc60 thru 0x0381c5 (0xc566 bytes) 000060 1763402718.311966 MIRROR_RESYNC 3 mirrors 000063 1763402719.009258 write 0x025397 thru 0x028bac (0x3816 bytes) 000069 1763402719.073247 trunc from 0x0381c6 to 0x023593 (0x14c32 bytes) 000070 1763402719.119551 read 0x01ccfa thru 0x01ec90 (0x1f97 bytes) 000075 1763402719.126891 mapread 0x004784 thru 0x009ee6 (0x5763 bytes) 000076 1763402719.138063 read 0x015bda thru 0x023593 (0xd9ba bytes) 000077 1763402719.159531 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x23594): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23594): Mirror 3: 0x54241300 (stale) Mirror 4: 0x54241300 (stale) CRC-32 checksum value for chunk [0, 0x23594): Mirror 1: 0x54241300 Mirror 5: 0x54241300 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23594: skipped 000079 1763402719.822642 read 0x022b52 thru 0x023593 (0xa42 bytes) 000080 1763402719.832666 mapwrite 0x03ff4c thru 0x03ffff (0xb4 bytes) 000083 1763402720.028938 trunc from 0x040000 to 0x009f38 (0x360c7 bytes) 000084 1763402720.052634 mapwrite 0x033006 thru 0x03567f (0x267a bytes) 000085 1763402720.143944 trunc from 0x035680 to 0x00ce3f (0x28840 bytes) 000086 1763402720.160569 trunc from 0x00ce40 to 0x0264dc (0x1969d bytes) 000087 1763402720.185444 mapread 0x0062b1 thru 0x00c8b2 (0x6602 bytes) 000088 1763402720.195359 read 0x00fad7 thru 0x00fe97 (0x3c1 bytes) 000089 1763402720.195997 read 0x024b74 thru 0x0264dc (0x1969 bytes) 000091 1763402720.215544 trunc from 0x0264dd to 0x00f66b (0x16e71 bytes) 000092 1763402720.243072 MIRROR_RESYNC 4 mirrors 000093 1763402721.068915 MIRROR_RESYNC 4 mirrors 000097 1763402721.369230 mapread 0x000111 thru 0x0047b7 (0x46a7 bytes) 000098 1763402721.392921 mapread 0x00f226 thru 0x00f66b (0x446 bytes) 000099 1763402721.393422 read 0x00ac28 thru 0x00cdc1 (0x219a bytes) 000100 1763402721.393694 read 0x0040b6 thru 0x00f66b (0xb5b6 bytes) 000101 1763402721.394023 write 0x010138 thru 0x014c9a (0x4b63 bytes) 000104 1763402721.468058 mapread 0x004369 thru 0x0082da (0x3f72 bytes) 000105 1763402721.480956 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x14c9b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14c9b): Mirror 3: 0x9de41540 (stale) Mirror 4: 0x9de41540 (stale) Mirror 5: 0x9de41540 (stale) CRC-32 checksum value for chunk [0, 0x14c9b): Mirror 1: 0xb45d713a Mirror 6: 0xb45d713a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14c9b: skipped 000107 1763402722.237635 mapwrite 0x03a3ce thru 0x03ffff (0x5c32 bytes) 000108 1763402722.403381 MIRROR_RESYNC 5 mirrors 000112 1763402723.101650 trunc from 0x040000 to 0x036a06 (0x95f9 bytes) 000114 1763402723.192757 mapwrite 0x0268ff thru 0x035259 (0xe95b bytes) 000115 1763402723.270531 mapwrite 0x01645f thru 0x024703 (0xe2a5 bytes) 000121 1763402723.342136 mapwrite 0x011949 thru 0x018a94 (0x714c bytes) 000122 1763402723.380886 mapwrite 0x006cdf thru 0x014971 (0xdc93 bytes) 000123 1763402723.435735 mapwrite 0x005de2 thru 0x010e50 (0xb06f bytes) 000124 1763402723.473858 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36a07): Mirror 3: 0x3182a9f4 (stale) Mirror 4: 0x3182a9f4 (stale) Mirror 5: 0x3182a9f4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36a07): Mirror 3: 0x3182a9f4 (stale) Mirror 4: 0x3182a9f4 (stale) Mirror 5: 0x3182a9f4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36a07: skipped 000125 1763402723.774068 mapread 0x030181 thru 0x035b5b (0x59db bytes) 000126 1763402723.799517 read 0x030d02 thru 0x034e33 (0x4132 bytes) 000128 1763402723.801107 write 0x01c32a thru 0x02b33c (0xf013 bytes) 000130 1763402723.809989 read 0x02e42a thru 0x036a06 (0x85dd bytes) 000131 1763402723.810611 mapwrite 0x02ffdf thru 0x03d764 (0xd786 bytes) 000132 1763402723.896546 MIRROR_RESYNC 4 mirrors 000133 1763402724.795752 write 0x001505 thru 0x004391 (0x2e8d bytes) 000134 1763402724.929946 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3d765): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d765): Mirror 3: 0x8ade99f (stale) Mirror 4: 0x8ade99f (stale) Mirror 5: 0x8ade99f (stale) CRC-32 checksum value for chunk [0, 0x3d765): Mirror 1: 0xb25ec3f4 Mirror 6: 0xb25ec3f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d765: skipped 000135 1763402725.847003 mapread 0x01e1b8 thru 0x02849c (0xa2e5 bytes) 000136 1763402725.881980 write 0x034560 thru 0x038f9e (0x4a3f bytes) 000137 1763402725.957455 read 0x015643 thru 0x01d07b (0x7a39 bytes) 000139 1763402725.971979 mapwrite 0x020dcb thru 0x03056c (0xf7a2 bytes) 000141 1763402726.055572 read 0x0047b5 thru 0x00f297 (0xaae3 bytes) 000144 1763402726.064762 mapwrite 0x024108 thru 0x03375f (0xf658 bytes) 000146 1763402726.114249 write 0x016a63 thru 0x02523d (0xe7db bytes) 000147 1763402726.115515 read 0x0280c8 thru 0x02c262 (0x419b bytes) 000149 1763402726.120312 MIRROR_RESYNC 5 mirrors 000152 1763402726.907882 mapread 0x022062 thru 0x0289c5 (0x6964 bytes) 000153 1763402726.924920 mapread 0x01e84e thru 0x02cb18 (0xe2cb bytes) 000154 1763402726.925521 read 0x001c03 thru 0x004d6e (0x316c bytes) 000155 1763402726.929947 read 0x01ad06 thru 0x022091 (0x738c bytes) 000156 1763402726.939848 trunc from 0x03d765 to 0x0010cb (0x3c699 bytes) 000159 1763402727.035876 write 0x02178c thru 0x031313 (0xfb88 bytes) 000161 1763402727.038620 read 0x001230 thru 0x008a8a (0x785b bytes) 000162 1763402727.051956 mapread 0x00ac8e thru 0x014d5f (0xa0d2 bytes) 000164 1763402727.053668 mapread 0x003921 thru 0x00d6e6 (0x9dc6 bytes) 000165 1763402727.060658 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x31314): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31314): Mirror 3: 0x54dbd5b1 (stale) Mirror 4: 0x54dbd5b1 (stale) Mirror 5: 0x54dbd5b1 (stale) Mirror 6: 0x54dbd5b1 (stale) CRC-32 checksum value for chunk [0, 0x31314): Mirror 1: 0x9b6c453e Mirror 7: 0x9b6c453e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31314: skipped 000166 1763402727.780714 mapread 0x006df0 thru 0x01354d (0xc75e bytes) 000167 1763402727.807708 write 0x000fdd thru 0x00b030 (0xa054 bytes) 000170 1763402727.897132 mapwrite 0x015dab thru 0x01a177 (0x43cd bytes) 000171 1763402727.974616 mapwrite 0x015853 thru 0x01e057 (0x8805 bytes) 000172 1763402728.053006 read 0x01c8aa thru 0x024d06 (0x845d bytes) 000173 1763402728.053645 mapread 0x023e80 thru 0x02769c (0x381d bytes) 000175 1763402728.063543 write 0x039b34 thru 0x03ad7f (0x124c bytes) 000177 1763402728.065705 trunc from 0x03ad80 to 0x03e756 (0x39d7 bytes) 000178 1763402728.108277 trunc from 0x03e757 to 0x026715 (0x18041 bytes) 000179 1763402728.135138 MIRROR_RESYNC 6 mirrors 000180 1763402729.196262 trunc from 0x026716 to 0x002a7c (0x23c99 bytes) 000182 1763402729.274355 read 0x0028ee thru 0x002a7c (0x18f bytes) 000183 1763402729.280844 mapwrite 0x01245d thru 0x01b5ec (0x9190 bytes) 000190 1763402729.392492 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b5ed): Mirror 4: 0x48f0d37b (stale) Mirror 5: 0x48f0d37b (stale) Mirror 6: 0x48f0d37b (stale) Mirror 7: 0x48f0d37b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b5ed): Mirror 4: 0x48f0d37b (stale) Mirror 5: 0x48f0d37b (stale) Mirror 6: 0x48f0d37b (stale) Mirror 7: 0x48f0d37b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b5ed: skipped 000193 1763402729.709332 mapwrite 0x007069 thru 0x00cb34 (0x5acc bytes) 000197 1763402729.777140 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1b5ed): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b5ed): Mirror 4: 0x48f0d37b (stale) Mirror 5: 0x48f0d37b (stale) Mirror 6: 0x48f0d37b (stale) Mirror 7: 0x48f0d37b (stale) CRC-32 checksum value for chunk [0, 0x1b5ed): Mirror 1: 0xa6f0cf5b Mirror 8: 0xa6f0cf5b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b5ed: skipped 000198 1763402730.476963 write 0x0339ae thru 0x033e90 (0x4e3 bytes) 000199 1763402730.544053 trunc from 0x033e91 to 0x00c3c9 (0x27ac7 bytes) 000200 1763402730.589981 write 0x024b2d thru 0x032737 (0xdc0b bytes) 000202 1763402730.591895 read 0x01354e thru 0x01f6f4 (0xc1a7 bytes) 000203 1763402730.610098 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x32738): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32738): Mirror 4: 0x60605e9e (stale) Mirror 5: 0x60605e9e (stale) Mirror 6: 0x60605e9e (stale) Mirror 7: 0x60605e9e (stale) Mirror 8: 0x2537ff95 (stale) CRC-32 checksum value for chunk [0, 0x32738): Mirror 1: 0x5f8f3f1b Mirror 9: 0x5f8f3f1b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32738: skipped 000204 1763402731.194553 mapread 0x02343e thru 0x032737 (0xf2fa bytes) 000205 1763402731.221669 read 0x019ddc thru 0x024327 (0xa54c bytes) 000206 1763402731.232132 mapwrite 0x003051 thru 0x009e23 (0x6dd3 bytes) 000207 1763402731.437338 mapwrite 0x01c451 thru 0x02144f (0x4fff bytes) 000208 1763402731.550386 read 0x025ec8 thru 0x02c2f5 (0x642e bytes) 000209 1763402731.576625 write 0x0069f6 thru 0x012977 (0xbf82 bytes) 000210 1763402731.577742 mapwrite 0x01c95a thru 0x023c52 (0x72f9 bytes) 000211 1763402731.663539 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x32738): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32738): Mirror 4: 0x60605e9e (stale) Mirror 5: 0x60605e9e (stale) Mirror 6: 0x60605e9e (stale) Mirror 7: 0x60605e9e (stale) Mirror 8: 0x2537ff95 (stale) Mirror 9: 0x5f8f3f1b (stale) CRC-32 checksum value for chunk [0, 0x32738): Mirror 1: 0x553e38d Mirror 10: 0x553e38d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32738: skipped 000212 1763402732.450130 mapread 0x032567 thru 0x032737 (0x1d1 bytes) 000214 1763402732.491909 mapwrite 0x0286a7 thru 0x03158d (0x8ee7 bytes) 000216 1763402732.730923 trunc from 0x032738 to 0x028e81 (0x98b6 bytes) 000217 1763402732.767656 write 0x01af64 thru 0x024400 (0x949d bytes) 000219 1763402732.796522 write 0x030ec1 thru 0x03ca44 (0xbb84 bytes) 000220 1763402732.801645 write 0x003e68 thru 0x009091 (0x522a bytes) 000221 1763402732.814525 mapwrite 0x00b459 thru 0x0115bd (0x6165 bytes) 000225 1763402732.889634 mapwrite 0x01c70d thru 0x024400 (0x7cf4 bytes) 000226 1763402732.938035 MIRROR_RESYNC 8 mirrors 000227 1763402733.997910 write 0x000fc8 thru 0x0038da (0x2913 bytes) 000228 1763402734.061577 mapwrite 0x01d8af thru 0x02c842 (0xef94 bytes) 000231 1763402734.143949 write 0x03881a thru 0x03ffff (0x77e6 bytes) 000232 1763402734.152189 mapread 0x02d3dc thru 0x039ae7 (0xc70c bytes) 000233 1763402734.165870 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x7b6b4449 (stale) Mirror 5: 0x7b6b4449 (stale) Mirror 6: 0x7b6b4449 (stale) Mirror 7: 0x7b6b4449 (stale) Mirror 8: 0x7b6b4449 (stale) Mirror 9: 0x7b6b4449 (stale) Mirror 10: 0x7b6b4449 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xeb44e886 Mirror 11: 0xeb44e886 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000235 1763402735.142739 mapwrite 0x0221c4 thru 0x02449a (0x22d7 bytes) 000236 1763402735.311939 trunc from 0x040000 to 0x01055f (0x2faa0 bytes) 000237 1763402735.350047 trunc from 0x010560 to 0x02c70f (0x1c1b0 bytes) 000238 1763402735.364756 read 0x0257fc thru 0x027f6d (0x2772 bytes) 000239 1763402735.373123 mapread 0x022a22 thru 0x02c70f (0x9cee bytes) 000241 1763402735.390542 mapread 0x00faae thru 0x016535 (0x6a88 bytes) 000243 1763402735.405447 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c710): Mirror 4: 0x7e1823e6 (stale) Mirror 5: 0x7e1823e6 (stale) Mirror 7: 0x7e1823e6 (stale) Mirror 8: 0x7e1823e6 (stale) Mirror 9: 0x7e1823e6 (stale) Mirror 10: 0x7e1823e6 (stale) Mirror 11: 0x8aab5fc6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c710): Mirror 4: 0x7e1823e6 (stale) Mirror 5: 0x7e1823e6 (stale) Mirror 7: 0x7e1823e6 (stale) Mirror 8: 0x7e1823e6 (stale) Mirror 9: 0x7e1823e6 (stale) Mirror 10: 0x7e1823e6 (stale) Mirror 11: 0x8aab5fc6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c710: skipped 000245 1763402735.823284 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c710): Mirror 4: 0x7e1823e6 (stale) Mirror 5: 0x7e1823e6 (stale) Mirror 7: 0x7e1823e6 (stale) Mirror 8: 0x7e1823e6 (stale) Mirror 9: 0x7e1823e6 (stale) Mirror 11: 0x8aab5fc6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c710): Mirror 4: 0x7e1823e6 (stale) Mirror 5: 0x7e1823e6 (stale) Mirror 7: 0x7e1823e6 (stale) Mirror 8: 0x7e1823e6 (stale) Mirror 9: 0x7e1823e6 (stale) Mirror 11: 0x8aab5fc6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c710: skipped 000247 1763402736.210588 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2c710): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c710): Mirror 4: 0x7e1823e6 (stale) Mirror 5: 0x7e1823e6 (stale) Mirror 7: 0x7e1823e6 (stale) Mirror 8: 0x7e1823e6 (stale) Mirror 9: 0x7e1823e6 (stale) Mirror 11: 0x8aab5fc6 (stale) CRC-32 checksum value for chunk [0, 0x2c710): Mirror 1: 0xcabdd58d Mirror 12: 0xcabdd58d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c710: skipped 000248 1763402736.974629 mapread 0x021729 thru 0x0238ed (0x21c5 bytes) 000251 1763402737.018354 mapwrite 0x01bc78 thru 0x023a2b (0x7db4 bytes) 000252 1763402737.220485 mapread 0x00b24b thru 0x018f3d (0xdcf3 bytes) 000253 1763402737.238389 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c710): Mirror 4: 0x7e1823e6 (stale) Mirror 5: 0x7e1823e6 (stale) Mirror 7: 0x7e1823e6 (stale) Mirror 8: 0x7e1823e6 (stale) Mirror 9: 0x7e1823e6 (stale) Mirror 11: 0x8aab5fc6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c710): Mirror 4: 0x7e1823e6 (stale) Mirror 5: 0x7e1823e6 (stale) Mirror 7: 0x7e1823e6 (stale) Mirror 8: 0x7e1823e6 (stale) Mirror 9: 0x7e1823e6 (stale) Mirror 11: 0x8aab5fc6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c710: skipped 000254 1763402737.678614 mapwrite 0x03dce8 thru 0x03ffff (0x2318 bytes) 000255 1763402737.781131 MIRROR_RESYNC 7 mirrors 000257 1763402738.924687 mapread 0x026a62 thru 0x02e4bd (0x7a5c bytes) 000258 1763402738.941708 MIRROR_EXTEND to 8 mirrors 000259 1763402739.570109 trunc from 0x040000 to 0x03680a (0x97f5 bytes) 000260 1763402739.675349 trunc from 0x03680b to 0x0337cd (0x303d bytes) 000262 1763402739.708379 MIRROR_RESYNC 8 mirrors 000264 1763402740.861277 trunc from 0x0337ce to 0x021c23 (0x11baa bytes) 000269 1763402740.946567 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x21c24): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21c24): Mirror 4: 0x6f09d3b1 (stale) Mirror 5: 0x6f09d3b1 (stale) Mirror 7: 0x6f09d3b1 (stale) Mirror 8: 0x6f09d3b1 (stale) Mirror 9: 0x6f09d3b1 (stale) Mirror 11: 0x6f09d3b1 (stale) Mirror 12: 0x6f09d3b1 (stale) CRC-32 checksum value for chunk [0, 0x21c24): Mirror 1: 0x6f09d3b1 Mirror 13: 0x6f09d3b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21c24: skipped 000270 1763402741.819438 mapwrite 0x01f27e thru 0x02f148 (0xfecb bytes) 000271 1763402742.103834 trunc from 0x02f149 to 0x035664 (0x651c bytes) 000276 1763402742.139022 MIRROR_RESYNC 9 mirrors 000277 1763402743.326105 mapread 0x009271 thru 0x01138b (0x811b bytes) 000278 1763402743.341283 write 0x0373f4 thru 0x03ffff (0x8c0c bytes) 000281 1763402743.416729 trunc from 0x040000 to 0x0037ae (0x3c851 bytes) 000282 1763402743.443125 MIRROR_RESYNC 9 mirrors 000283 1763402744.364478 trunc from 0x0037af to 0x032aea (0x2f33c bytes) 000285 1763402744.425193 mapread 0x0030b3 thru 0x010823 (0xd771 bytes) 000286 1763402744.449318 read 0x01a1c0 thru 0x01ec79 (0x4aba bytes) 000289 1763402744.480409 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32aeb): Mirror 4: 0x6de7c9a4 (stale) Mirror 5: 0x6de7c9a4 (stale) Mirror 7: 0x6de7c9a4 (stale) Mirror 8: 0x6de7c9a4 (stale) Mirror 11: 0x6de7c9a4 (stale) Mirror 12: 0x6de7c9a4 (stale) Mirror 13: 0x6de7c9a4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32aeb): Mirror 4: 0x6de7c9a4 (stale) Mirror 5: 0x6de7c9a4 (stale) Mirror 7: 0x6de7c9a4 (stale) Mirror 8: 0x6de7c9a4 (stale) Mirror 11: 0x6de7c9a4 (stale) Mirror 12: 0x6de7c9a4 (stale) Mirror 13: 0x6de7c9a4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32aeb: skipped 000292 1763402744.891114 mapwrite 0x01e002 thru 0x02b4a9 (0xd4a8 bytes) 000295 1763402744.970247 read 0x02d680 thru 0x032aea (0x546b bytes) 000296 1763402744.978448 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32aeb): Mirror 4: 0x6de7c9a4 (stale) Mirror 5: 0x6de7c9a4 (stale) Mirror 7: 0x6de7c9a4 (stale) Mirror 8: 0x6de7c9a4 (stale) Mirror 11: 0x6de7c9a4 (stale) Mirror 13: 0x6de7c9a4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32aeb): Mirror 4: 0x6de7c9a4 (stale) Mirror 5: 0x6de7c9a4 (stale) Mirror 7: 0x6de7c9a4 (stale) Mirror 8: 0x6de7c9a4 (stale) Mirror 11: 0x6de7c9a4 (stale) Mirror 13: 0x6de7c9a4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32aeb: skipped 000297 1763402745.307990 mapread 0x00ed09 thru 0x01945e (0xa756 bytes) 000298 1763402745.331545 mapread 0x014613 thru 0x023427 (0xee15 bytes) 000299 1763402745.339264 read 0x00195d thru 0x00aa5c (0x9100 bytes) 000300 1763402745.346752 mapwrite 0x036642 thru 0x03fed8 (0x9897 bytes) 000301 1763402745.434025 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fed9): Mirror 4: 0x68600cf4 (stale) Mirror 5: 0x68600cf4 (stale) Mirror 7: 0x68600cf4 (stale) Mirror 8: 0x68600cf4 (stale) Mirror 13: 0x68600cf4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fed9): Mirror 4: 0x68600cf4 (stale) Mirror 5: 0x68600cf4 (stale) Mirror 7: 0x68600cf4 (stale) Mirror 8: 0x68600cf4 (stale) Mirror 13: 0x68600cf4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fed9: skipped 000302 1763402745.754588 read 0x0154fc thru 0x02478c (0xf291 bytes) 000303 1763402745.763355 trunc from 0x03fed9 to 0x024199 (0x1bd3f bytes) 000305 1763402745.783730 trunc from 0x02419a to 0x02c2a0 (0x8107 bytes) 000306 1763402745.800522 write 0x0043a4 thru 0x004b40 (0x79d bytes) 000307 1763402745.805523 mapwrite 0x01fea6 thru 0x02305f (0x31ba bytes) 000308 1763402745.846129 mapread 0x0152f8 thru 0x017481 (0x218a bytes) 000309 1763402745.847061 trunc from 0x02c2a1 to 0x016d75 (0x1552b bytes) 000314 1763402745.864542 trunc from 0x016d76 to 0x03bfc2 (0x2524d bytes) 000315 1763402745.880641 mapwrite 0x013e5c thru 0x02226f (0xe414 bytes) 000317 1763402745.942089 write 0x03edb8 thru 0x03ffff (0x1248 bytes) 000318 1763402745.943022 mapwrite 0x02f3b4 thru 0x03906d (0x9cba bytes) 000319 1763402746.005880 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x2ac1a5e0 (stale) Mirror 5: 0x2ac1a5e0 (stale) Mirror 7: 0x2ac1a5e0 (stale) Mirror 8: 0x2ac1a5e0 (stale) Mirror 13: 0x2ac1a5e0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x64fb05b Mirror 14: 0x64fb05b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000320 1763402746.671359 read 0x02be3f thru 0x038347 (0xc509 bytes) 000321 1763402746.693059 read 0x011dc4 thru 0x01fa73 (0xdcb0 bytes) 000323 1763402746.720635 trunc from 0x040000 to 0x01f992 (0x2066d bytes) 000324 1763402746.808962 trunc from 0x01f993 to 0x000d2d (0x1ec65 bytes) 000326 1763402746.825082 write 0x016ea2 thru 0x017fbb (0x111a bytes) 000327 1763402746.826061 mapread 0x002502 thru 0x00a66f (0x816e bytes) 000328 1763402746.865154 trunc from 0x017fbc to 0x019509 (0x154e bytes) 000330 1763402746.896757 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1950a): Mirror 5: 0x4b606423 (stale) Mirror 7: 0x4b606423 (stale) Mirror 8: 0x4b606423 (stale) Mirror 13: 0x4b606423 (stale) Mirror 14: 0x25825fb9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1950a): Mirror 5: 0x4b606423 (stale) Mirror 7: 0x4b606423 (stale) Mirror 8: 0x4b606423 (stale) Mirror 13: 0x4b606423 (stale) Mirror 14: 0x25825fb9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1950a: skipped 000334 1763402747.382956 write 0x03c577 thru 0x03ffff (0x3a89 bytes) 000336 1763402747.386268 write 0x034d73 thru 0x039a7a (0x4d08 bytes) 000337 1763402747.407061 write 0x01581e thru 0x01f3fc (0x9bdf bytes) 000339 1763402747.430130 mapread 0x0151f6 thru 0x023e58 (0xec63 bytes) 000340 1763402747.440435 mapwrite 0x023569 thru 0x025103 (0x1b9b bytes) 000343 1763402747.520868 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x2ac1a5e0 (stale) Mirror 7: 0x2ac1a5e0 (stale) Mirror 8: 0x2ac1a5e0 (stale) Mirror 13: 0x2ac1a5e0 (stale) Mirror 14: 0x64fb05b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1e1a9236 Mirror 15: 0x1e1a9236 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000344 1763402748.519062 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x2ac1a5e0 (stale) Mirror 7: 0x2ac1a5e0 (stale) Mirror 8: 0x2ac1a5e0 (stale) Mirror 13: 0x2ac1a5e0 (stale) Mirror 14: 0x64fb05b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1e1a9236 Mirror 15: 0x1e1a9236 Mirror 16: 0x1e1a9236 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000345 1763402749.321426 write 0x037f90 thru 0x03ffff (0x8070 bytes) 000346 1763402749.403097 trunc from 0x040000 to 0x00d74a (0x328b5 bytes) 000348 1763402749.453201 mapwrite 0x004d89 thru 0x0097e8 (0x4a60 bytes) 000350 1763402749.505439 mapread 0x0034dd thru 0x00d74a (0xa26e bytes) 000355 1763402749.508056 read 0x0077e8 thru 0x00d74a (0x5f63 bytes) 000356 1763402749.508622 write 0x00db9a thru 0x01d700 (0xfb67 bytes) 000357 1763402749.510098 read 0x0030ea thru 0x003abc (0x9d3 bytes) 000361 1763402749.515321 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1d701): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d701): Mirror 5: 0xe72ac722 (stale) Mirror 7: 0xe72ac722 (stale) Mirror 8: 0xe72ac722 (stale) Mirror 13: 0xe72ac722 (stale) Mirror 14: 0x7fb7be1c (stale) Mirror 15: 0xc087c2a4 (stale) Mirror 16: 0xc087c2a4 (stale) CRC-32 checksum value for chunk [0, 0x1d701): Mirror 1: 0x59578db Mirror 17: 0x59578db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d701: skipped 000362 1763402750.208093 mapread 0x00ec75 thru 0x01b1c2 (0xc54e bytes) 000363 1763402750.254363 trunc from 0x01d701 to 0x00ca57 (0x10ca9 bytes) 000364 1763402750.345244 read 0x000a23 thru 0x00ca57 (0xc035 bytes) 000365 1763402750.352036 mapread 0x009175 thru 0x00ca57 (0x38e3 bytes) 000366 1763402750.352837 mapread 0x00c150 thru 0x00ca57 (0x908 bytes) 000367 1763402750.353449 mapwrite 0x018c6d thru 0x025622 (0xc9b6 bytes) 000368 1763402750.442774 trunc from 0x025623 to 0x02a988 (0x5366 bytes) 000369 1763402750.460523 read 0x029ae1 thru 0x02a988 (0xea8 bytes) 000371 1763402750.469560 mapwrite 0x03b07c thru 0x03b7bb (0x740 bytes) 000372 1763402750.531349 write 0x009f21 thru 0x01564e (0xb72e bytes) 000373 1763402750.532558 write 0x03faef thru 0x03ffff (0x511 bytes) 000374 1763402750.533312 mapwrite 0x01729c thru 0x022c5f (0xb9c4 bytes) 000376 1763402750.592514 trunc from 0x040000 to 0x01b947 (0x246b8 bytes) 000378 1763402750.615953 MIRROR_RESYNC 9 mirrors 000381 1763402751.693882 read 0x00298f thru 0x006acc (0x413e bytes) 000382 1763402751.707314 write 0x03698e thru 0x03ffff (0x9672 bytes) 000384 1763402751.760667 read 0x016427 thru 0x023d1a (0xd8f4 bytes) 000385 1763402751.797032 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe27ad12e (stale) Mirror 7: 0xe27ad12e (stale) Mirror 8: 0xe27ad12e (stale) Mirror 13: 0xe27ad12e (stale) Mirror 14: 0xe27ad12e (stale) Mirror 15: 0xe27ad12e (stale) Mirror 17: 0xe27ad12e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe27ad12e (stale) Mirror 7: 0xe27ad12e (stale) Mirror 8: 0xe27ad12e (stale) Mirror 13: 0xe27ad12e (stale) Mirror 14: 0xe27ad12e (stale) Mirror 15: 0xe27ad12e (stale) Mirror 17: 0xe27ad12e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000386 1763402752.352258 read 0x02bb2a thru 0x03ba49 (0xff20 bytes) 000387 1763402752.363711 trunc from 0x040000 to 0x01c01b (0x23fe4 bytes) 000388 1763402752.399072 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1c01c): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c01c): Mirror 5: 0xc9381fa7 (stale) Mirror 7: 0xc9381fa7 (stale) Mirror 8: 0xc9381fa7 (stale) Mirror 13: 0xc9381fa7 (stale) Mirror 14: 0xc9381fa7 (stale) Mirror 15: 0xc9381fa7 (stale) Mirror 17: 0xc9381fa7 (stale) CRC-32 checksum value for chunk [0, 0x1c01c): Mirror 1: 0xc9381fa7 Mirror 18: 0xc9381fa7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c01c: skipped 000389 1763402753.086999 write 0x02d442 thru 0x033d17 (0x68d6 bytes) 000390 1763402753.152738 trunc from 0x033d18 to 0x0028aa (0x3146d bytes) 000391 1763402753.202376 trunc from 0x0028ab to 0x01fd17 (0x1d46d bytes) 000393 1763402753.237835 trunc from 0x01fd18 to 0x038a79 (0x18d62 bytes) 000394 1763402753.266193 mapread 0x005bc1 thru 0x0154fc (0xf93c bytes) 000395 1763402753.294905 trunc from 0x038a7a to 0x01ab16 (0x1df63 bytes) 000396 1763402753.312711 trunc from 0x01ab17 to 0x035d0c (0x1b1f6 bytes) 000397 1763402753.335607 write 0x03b608 thru 0x03ffff (0x49f8 bytes) 000399 1763402753.337006 write 0x031ea6 thru 0x03958a (0x76e5 bytes) 000401 1763402753.349493 trunc from 0x040000 to 0x039337 (0x6cc8 bytes) 000403 1763402753.369114 write 0x00ddc3 thru 0x01d646 (0xf884 bytes) 000406 1763402753.377865 mapread 0x029de8 thru 0x02ab07 (0xd20 bytes) 000408 1763402753.390054 trunc from 0x039338 to 0x037f42 (0x13f5 bytes) 000409 1763402753.404194 mapwrite 0x02374d thru 0x02680a (0x30be bytes) 000410 1763402753.465439 mapwrite 0x01fb99 thru 0x02f7cf (0xfc37 bytes) 000411 1763402753.513438 mapread 0x007479 thru 0x013c7b (0xc803 bytes) 000413 1763402753.514872 trunc from 0x037f43 to 0x02a9c9 (0xd579 bytes) 000415 1763402753.533953 write 0x001942 thru 0x006b81 (0x5240 bytes) 000417 1763402753.535285 mapread 0x012dbe thru 0x015949 (0x2b8c bytes) 000419 1763402753.536417 write 0x001c31 thru 0x00c6ea (0xaaba bytes) 000421 1763402753.538131 mapread 0x024719 thru 0x02a9c9 (0x62b1 bytes) 000424 1763402753.539952 mapread 0x025928 thru 0x02a9c9 (0x50a2 bytes) 000426 1763402753.541209 read 0x00587e thru 0x015535 (0xfcb8 bytes) 000427 1763402753.541787 trunc from 0x02a9ca to 0x01f71c (0xb2ad bytes) 000428 1763402753.570924 read 0x00197d thru 0x00fc45 (0xe2c9 bytes) 000431 1763402753.573320 trunc from 0x01f71d to 0x024929 (0x520d bytes) 000433 1763402753.590467 mapread 0x01304a thru 0x017c61 (0x4c18 bytes) 000434 1763402753.591404 mapread 0x006d9c thru 0x0102aa (0x950f bytes) 000436 1763402753.592738 mapread 0x00356d thru 0x003ad5 (0x569 bytes) 000441 1763402753.600446 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2492a): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2492a): Mirror 5: 0x55873b85 (stale) Mirror 7: 0x55873b85 (stale) Mirror 8: 0x55873b85 (stale) Mirror 13: 0x55873b85 (stale) Mirror 14: 0x55873b85 (stale) Mirror 15: 0x55873b85 (stale) Mirror 17: 0x55873b85 (stale) Mirror 18: 0x55873b85 (stale) CRC-32 checksum value for chunk [0, 0x2492a): Mirror 1: 0x33d888bd Mirror 19: 0x33d888bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2492a: skipped 000443 1763402754.367583 mapread 0x023f79 thru 0x024929 (0x9b1 bytes) 000444 1763402754.393999 write 0x0277f5 thru 0x02eb36 (0x7342 bytes) 000445 1763402754.479178 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2eb37): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2eb37): Mirror 5: 0xe075e90b (stale) Mirror 7: 0xe075e90b (stale) Mirror 8: 0xe075e90b (stale) Mirror 13: 0xe075e90b (stale) Mirror 14: 0xe075e90b (stale) Mirror 15: 0xe075e90b (stale) Mirror 17: 0xe075e90b (stale) Mirror 18: 0xe075e90b (stale) Mirror 19: 0xd9d65a6e (stale) CRC-32 checksum value for chunk [0, 0x2eb37): Mirror 1: 0xd7f080d6 Mirror 20: 0xd7f080d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2eb37: skipped 000447 1763402755.373917 write 0x02d9b7 thru 0x03874f (0xad99 bytes) 000448 1763402755.482268 write 0x00d3f7 thru 0x01ad1e (0xd928 bytes) 000450 1763402755.505211 mapwrite 0x021040 thru 0x029a29 (0x89ea bytes) 000451 1763402755.576832 mapread 0x022f54 thru 0x029712 (0x67bf bytes) 000453 1763402755.578087 mapread 0x0151c8 thru 0x0203fa (0xb233 bytes) 000454 1763402755.583661 MIRROR_RESYNC 11 mirrors 000455 1763402757.208207 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) 000456 1763402757.735383 mapwrite 0x002153 thru 0x00713e (0x4fec bytes) 000458 1763402757.971195 mapread 0x009e12 thru 0x0183eb (0xe5da bytes) 000459 1763402757.983682 mapread 0x01ebdc thru 0x0258d7 (0x6cfc bytes) 000460 1763402757.995001 write 0x01a4ba thru 0x01ee26 (0x496d bytes) 000464 1763402757.997199 read 0x03580d thru 0x035e5a (0x64e bytes) 000465 1763402758.014541 write 0x009053 thru 0x00cb41 (0x3aef bytes) 000466 1763402758.015787 mapwrite 0x00c525 thru 0x018691 (0xc16d bytes) 000467 1763402758.088839 read 0x02f857 thru 0x031ccf (0x2479 bytes) 000468 1763402758.105286 MIRROR_RESYNC 10 mirrors 000469 1763402759.564117 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) 000471 1763402759.903776 mapread 0x0300c6 thru 0x038167 (0x80a2 bytes) 000473 1763402759.921015 mapwrite 0x0306c4 thru 0x0318c9 (0x1206 bytes) 000475 1763402760.089948 write 0x01ceb3 thru 0x01d46a (0x5b8 bytes) 000476 1763402760.101899 mapread 0x022359 thru 0x02f5df (0xd287 bytes) 000477 1763402760.108226 mapread 0x028e3e thru 0x02ecb7 (0x5e7a bytes) 000479 1763402760.113162 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38750): Mirror 8: 0x8e5a31af (stale) Mirror 13: 0x8e5a31af (stale) Mirror 14: 0x8e5a31af (stale) Mirror 17: 0x8e5a31af (stale) Mirror 18: 0x8e5a31af (stale) Mirror 19: 0x8e5a31af (stale) Mirror 20: 0x8e5a31af (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38750): Mirror 8: 0x8e5a31af (stale) Mirror 13: 0x8e5a31af (stale) Mirror 14: 0x8e5a31af (stale) Mirror 17: 0x8e5a31af (stale) Mirror 18: 0x8e5a31af (stale) Mirror 19: 0x8e5a31af (stale) Mirror 20: 0x8e5a31af (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38750: skipped 000481 1763402760.668518 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x38750): Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38750): Mirror 8: 0x8e5a31af (stale) Mirror 13: 0x8e5a31af (stale) Mirror 14: 0x8e5a31af (stale) Mirror 17: 0x8e5a31af (stale) Mirror 18: 0x8e5a31af (stale) Mirror 19: 0x8e5a31af (stale) Mirror 20: 0x8e5a31af (stale) CRC-32 checksum value for chunk [0, 0x38750): Mirror 1: 0x480b754e Mirror 21: 0x480b754e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38750: skipped 000482 1763402761.550724 read 0x0084a0 thru 0x008f84 (0xae5 bytes) 000483 1763402761.565028 trunc from 0x038750 to 0x00bb24 (0x2cc2b bytes) 000486 1763402761.660758 mapwrite 0x03b91c thru 0x03ffff (0x46e4 bytes) 000487 1763402761.767967 read 0x01cc6c thru 0x02710d (0xa4a2 bytes) 000490 1763402761.791838 mapread 0x01e372 thru 0x02770b (0x939a bytes) 000493 1763402761.797818 trunc from 0x040000 to 0x00277e (0x3d881 bytes) 000494 1763402761.818106 write 0x02db75 thru 0x038354 (0xa7e0 bytes) 000495 1763402761.820417 write 0x015909 thru 0x01c85f (0x6f57 bytes) 000496 1763402761.834722 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38355): Mirror 8: 0xa95b285a (stale) Mirror 13: 0xa95b285a (stale) Mirror 14: 0xa95b285a (stale) Mirror 18: 0xa95b285a (stale) Mirror 19: 0xa95b285a (stale) Mirror 20: 0xa95b285a (stale) Mirror 21: 0xe5b6bdc2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38355): Mirror 8: 0xa95b285a (stale) Mirror 13: 0xa95b285a (stale) Mirror 14: 0xa95b285a (stale) Mirror 18: 0xa95b285a (stale) Mirror 19: 0xa95b285a (stale) Mirror 20: 0xa95b285a (stale) Mirror 21: 0xe5b6bdc2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38355: skipped 000499 1763402762.253033 read 0x0286d1 thru 0x0376fd (0xf02d bytes) 000500 1763402762.265039 write 0x030024 thru 0x033c9c (0x3c79 bytes) 000502 1763402762.266343 mapwrite 0x023ebd thru 0x02aefd (0x7041 bytes) 000503 1763402762.325319 mapwrite 0x03c8e7 thru 0x03ffff (0x3719 bytes) 000504 1763402762.382459 trunc from 0x040000 to 0x013408 (0x2cbf7 bytes) 000506 1763402762.396780 trunc from 0x013409 to 0x025f12 (0x12b0a bytes) 000507 1763402762.421231 MIRROR_RESYNC 8 mirrors 000508 1763402763.477547 mapread 0x009a9a thru 0x01783b (0xdda2 bytes) 000509 1763402763.504759 mapwrite 0x00c0a3 thru 0x01310a (0x7068 bytes) 000510 1763402763.661722 mapwrite 0x00599e thru 0x00cd68 (0x73cb bytes) 000511 1763402763.706284 mapread 0x00af58 thru 0x00c1ef (0x1298 bytes) 000512 1763402763.707077 read 0x00dd11 thru 0x015142 (0x7432 bytes) 000513 1763402763.717274 MIRROR_RESYNC 8 mirrors 000514 1763402764.858042 mapwrite 0x00abac thru 0x010b61 (0x5fb6 bytes) 000516 1763402765.108270 read 0x0175e4 thru 0x023a3a (0xc457 bytes) 000517 1763402765.123335 mapwrite 0x02e8bf thru 0x03d7d5 (0xef17 bytes) 000521 1763402765.218055 read 0x027cdd thru 0x02cd0d (0x5031 bytes) 000525 1763402765.230681 MIRROR_RESYNC 8 mirrors 000526 1763402766.229036 mapwrite 0x013d95 thru 0x016249 (0x24b5 bytes) 000528 1763402766.427069 MIRROR_RESYNC 8 mirrors 000531 1763402767.469015 read 0x01bf22 thru 0x02bce3 (0xfdc2 bytes) 000532 1763402767.482671 write 0x03afb8 thru 0x03e8e5 (0x392e bytes) 000533 1763402767.539221 write 0x02673e thru 0x02f29a (0x8b5d bytes) 000534 1763402767.545771 mapwrite 0x01a15a thru 0x0276c7 (0xd56e bytes) 000536 1763402767.615720 mapread 0x035c7d thru 0x03e8e5 (0x8c69 bytes) 000538 1763402767.626923 read 0x018158 thru 0x01d801 (0x56aa bytes) 000541 1763402767.633965 write 0x02c779 thru 0x037dab (0xb633 bytes) 000542 1763402767.635771 write 0x0058f7 thru 0x00bdca (0x64d4 bytes) 000543 1763402767.644287 mapwrite 0x02e086 thru 0x03be58 (0xddd3 bytes) 000544 1763402767.719410 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3e8e6): Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e8e6): Mirror 8: 0x6a096b56 (stale) Mirror 13: 0x6a096b56 (stale) Mirror 14: 0x6a096b56 (stale) Mirror 18: 0x6a096b56 (stale) Mirror 19: 0x6a096b56 (stale) Mirror 20: 0x6a096b56 (stale) Mirror 21: 0x6a096b56 (stale) CRC-32 checksum value for chunk [0, 0x3e8e6): Mirror 1: 0x7003569c Mirror 22: 0x7003569c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e8e6: skipped 000547 1763402768.559603 write 0x017311 thru 0x01cfda (0x5cca bytes) 000548 1763402768.644742 mapwrite 0x007610 thru 0x010c32 (0x9623 bytes) 000549 1763402768.733247 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e8e6): Mirror 13: 0x6a096b56 (stale) Mirror 14: 0x6a096b56 (stale) Mirror 18: 0x6a096b56 (stale) Mirror 19: 0x6a096b56 (stale) Mirror 20: 0x6a096b56 (stale) Mirror 21: 0x6a096b56 (stale) Mirror 22: 0x7003569c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e8e6): Mirror 13: 0x6a096b56 (stale) Mirror 14: 0x6a096b56 (stale) Mirror 18: 0x6a096b56 (stale) Mirror 19: 0x6a096b56 (stale) Mirror 20: 0x6a096b56 (stale) Mirror 21: 0x6a096b56 (stale) Mirror 22: 0x7003569c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e8e6: skipped 000550 1763402769.268610 MIRROR_RESYNC 8 mirrors 000551 1763402770.461320 mapread 0x03b4af thru 0x03e8e5 (0x3437 bytes) 000552 1763402770.485307 write 0x01aeb1 thru 0x01ed95 (0x3ee5 bytes) 000553 1763402770.604268 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3e8e6): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e8e6): Mirror 13: 0x96ccc95c (stale) Mirror 14: 0x96ccc95c (stale) Mirror 18: 0x96ccc95c (stale) Mirror 19: 0x96ccc95c (stale) Mirror 20: 0x96ccc95c (stale) Mirror 21: 0x96ccc95c (stale) Mirror 22: 0x96ccc95c (stale) CRC-32 checksum value for chunk [0, 0x3e8e6): Mirror 1: 0x71d099a3 Mirror 23: 0x71d099a3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e8e6: skipped 000554 1763402771.363699 trunc from 0x03e8e6 to 0x02f9c9 (0xef1c bytes) 000556 1763402771.452240 trunc from 0x02f9ca to 0x02eb73 (0xe56 bytes) 000558 1763402771.481986 MIRROR_RESYNC 9 mirrors 000559 1763402772.654163 write 0x01e8da thru 0x02261c (0x3d43 bytes) 000563 1763402772.722307 read 0x011300 thru 0x01b529 (0xa22a bytes) 000564 1763402772.731769 mapread 0x005be9 thru 0x00d191 (0x75a9 bytes) 000565 1763402772.744433 write 0x01ffd2 thru 0x02451f (0x454e bytes) 000566 1763402772.756119 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2eb74): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2eb74): Mirror 13: 0xcdb91067 (stale) Mirror 14: 0xcdb91067 (stale) Mirror 18: 0xcdb91067 (stale) Mirror 19: 0xcdb91067 (stale) Mirror 20: 0xcdb91067 (stale) Mirror 21: 0xcdb91067 (stale) Mirror 22: 0xcdb91067 (stale) Mirror 23: 0xcdb91067 (stale) CRC-32 checksum value for chunk [0, 0x2eb74): Mirror 1: 0x2e73b11 Mirror 24: 0x2e73b11 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2eb74: skipped 000568 1763402773.542227 trunc from 0x02eb74 to 0x01de02 (0x10d71 bytes) 000569 1763402773.614410 mapwrite 0x035bce thru 0x03ffff (0xa432 bytes) 000571 1763402773.721125 mapread 0x03eb5d thru 0x03ffff (0x14a3 bytes) 000573 1763402773.722396 mapwrite 0x00737b thru 0x009670 (0x22f6 bytes) 000574 1763402773.785142 mapread 0x00f4da thru 0x01c891 (0xd3b8 bytes) 000577 1763402773.816152 trunc from 0x040000 to 0x03dc35 (0x23ca bytes) 000578 1763402773.836669 write 0x0009e3 thru 0x003226 (0x2844 bytes) 000582 1763402773.845252 mapread 0x01be49 thru 0x01e9b7 (0x2b6f bytes) 000583 1763402773.846081 write 0x0369d1 thru 0x03ffff (0x962f bytes) 000586 1763402773.855757 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xabd21146 (stale) Mirror 14: 0xabd21146 (stale) Mirror 18: 0xabd21146 (stale) Mirror 19: 0xabd21146 (stale) Mirror 20: 0xabd21146 (stale) Mirror 21: 0xabd21146 (stale) Mirror 22: 0xabd21146 (stale) Mirror 23: 0xabd21146 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xabd21146 (stale) Mirror 14: 0xabd21146 (stale) Mirror 18: 0xabd21146 (stale) Mirror 19: 0xabd21146 (stale) Mirror 20: 0xabd21146 (stale) Mirror 21: 0xabd21146 (stale) Mirror 22: 0xabd21146 (stale) Mirror 23: 0xabd21146 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000588 1763402774.272655 trunc from 0x040000 to 0x000c62 (0x3f39d bytes) 000589 1763402774.290548 mapread 0x0000c9 thru 0x000c62 (0xb9a bytes) 000593 1763402774.301080 trunc from 0x000c63 to 0x015f50 (0x152ee bytes) 000597 1763402774.320729 mapread 0x002f0e thru 0x004fc9 (0x20bc bytes) 000598 1763402774.342906 mapwrite 0x03030a thru 0x038c70 (0x8967 bytes) 000599 1763402774.468378 write 0x025775 thru 0x02838e (0x2c1a bytes) 000601 1763402774.486425 trunc from 0x038c71 to 0x01fd2e (0x18f42 bytes) 000603 1763402774.518441 read 0x01eb9e thru 0x01fd2e (0x1191 bytes) 000605 1763402774.519481 trunc from 0x01fd2f to 0x0007fc (0x1f532 bytes) 000606 1763402774.539434 mapwrite 0x01c1ba thru 0x02b0b2 (0xeef9 bytes) 000607 1763402774.678208 mapwrite 0x00745a thru 0x014703 (0xd2aa bytes) 000608 1763402774.789887 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2b0b3): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b0b3): Mirror 13: 0x420f704c (stale) Mirror 14: 0x420f704c (stale) Mirror 18: 0x420f704c (stale) Mirror 19: 0x420f704c (stale) Mirror 20: 0x420f704c (stale) Mirror 21: 0x420f704c (stale) Mirror 22: 0x420f704c (stale) Mirror 23: 0x420f704c (stale) CRC-32 checksum value for chunk [0, 0x2b0b3): Mirror 1: 0x3b856d69 Mirror 24: 0x3b856d69 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b0b3: skipped 000609 1763402775.727318 trunc from 0x02b0b3 to 0x02f2fb (0x4249 bytes) 000610 1763402775.811850 trunc from 0x02f2fc to 0x00f3d9 (0x1ff22 bytes) 000611 1763402775.850164 write 0x02d2ab thru 0x033ed7 (0x6c2d bytes) 000613 1763402775.853731 write 0x017f11 thru 0x01a2aa (0x239a bytes) 000615 1763402775.869909 trunc from 0x033ed8 to 0x019baf (0x1a328 bytes) 000616 1763402775.884043 trunc from 0x019bb0 to 0x02702c (0xd47d bytes) 000617 1763402775.912300 write 0x0011b8 thru 0x00e812 (0xd65b bytes) 000618 1763402775.922676 mapwrite 0x02f4d1 thru 0x03c94f (0xd47f bytes) 000621 1763402776.008139 trunc from 0x03c950 to 0x0348b4 (0x809b bytes) 000622 1763402776.025314 read 0x003de1 thru 0x004a99 (0xcb9 bytes) 000623 1763402776.025945 mapwrite 0x01aec5 thru 0x01c5fa (0x1736 bytes) 000625 1763402776.075738 mapwrite 0x01ffa1 thru 0x02c5d5 (0xc635 bytes) 000626 1763402776.158647 mapread 0x00a32d thru 0x00d341 (0x3015 bytes) 000628 1763402776.160461 mapwrite 0x00ed93 thru 0x01c443 (0xd6b1 bytes) 000629 1763402776.214470 mapread 0x033d24 thru 0x0348b4 (0xb91 bytes) 000630 1763402776.219963 MIRROR_RESYNC 10 mirrors 000631 1763402777.477741 mapread 0x00d64b thru 0x00f00a (0x19c0 bytes) 000632 1763402777.495361 trunc from 0x0348b5 to 0x03d7b3 (0x8eff bytes) 000636 1763402777.562354 read 0x02b168 thru 0x035984 (0xa81d bytes) 000637 1763402777.569596 mapwrite 0x0233d1 thru 0x031fa2 (0xebd2 bytes) 000638 1763402777.635531 mapwrite 0x03e92c thru 0x03ffff (0x16d4 bytes) 000640 1763402777.696153 write 0x01c0ff thru 0x02444e (0x8350 bytes) 000643 1763402777.704865 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x8f0710e7 (stale) Mirror 18: 0x8f0710e7 (stale) Mirror 19: 0x8f0710e7 (stale) Mirror 20: 0x8f0710e7 (stale) Mirror 21: 0x8f0710e7 (stale) Mirror 22: 0x8f0710e7 (stale) Mirror 23: 0x8f0710e7 (stale) Mirror 24: 0x8f0710e7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x8f0710e7 (stale) Mirror 18: 0x8f0710e7 (stale) Mirror 19: 0x8f0710e7 (stale) Mirror 20: 0x8f0710e7 (stale) Mirror 21: 0x8f0710e7 (stale) Mirror 22: 0x8f0710e7 (stale) Mirror 23: 0x8f0710e7 (stale) Mirror 24: 0x8f0710e7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000644 1763402778.187750 mapwrite 0x038deb thru 0x03ffff (0x7215 bytes) 000645 1763402778.259392 trunc from 0x040000 to 0x038729 (0x78d6 bytes) 000646 1763402778.302785 MIRROR_RESYNC 9 mirrors 000650 1763402779.492671 read 0x02f6ba thru 0x038729 (0x9070 bytes) 000651 1763402779.499520 mapwrite 0x002baf thru 0x0074b6 (0x4908 bytes) 000652 1763402779.651004 read 0x013d5c thru 0x01e2de (0xa583 bytes) 000653 1763402779.681557 trunc from 0x03872a to 0x0213f9 (0x17330 bytes) 000655 1763402779.716407 mapwrite 0x031b91 thru 0x033f45 (0x23b5 bytes) 000659 1763402779.797283 write 0x010104 thru 0x015a75 (0x5972 bytes) 000660 1763402779.798349 write 0x0187fe thru 0x01fc43 (0x7446 bytes) 000663 1763402779.800179 mapread 0x013a78 thru 0x015b44 (0x20cd bytes) 000664 1763402779.800962 mapread 0x000b00 thru 0x00c115 (0xb616 bytes) 000666 1763402779.802249 write 0x024ffb thru 0x02acf5 (0x5cfb bytes) 000668 1763402779.817474 mapwrite 0x0132b5 thru 0x01f07c (0xbdc8 bytes) 000669 1763402779.896298 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33f46): Mirror 14: 0xd4bfc518 (stale) Mirror 18: 0xd4bfc518 (stale) Mirror 20: 0xd4bfc518 (stale) Mirror 21: 0xd4bfc518 (stale) Mirror 22: 0xd4bfc518 (stale) Mirror 23: 0xd4bfc518 (stale) Mirror 24: 0xd4bfc518 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33f46): Mirror 14: 0xd4bfc518 (stale) Mirror 18: 0xd4bfc518 (stale) Mirror 20: 0xd4bfc518 (stale) Mirror 21: 0xd4bfc518 (stale) Mirror 22: 0xd4bfc518 (stale) Mirror 23: 0xd4bfc518 (stale) Mirror 24: 0xd4bfc518 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33f46: skipped 000670 1763402780.361618 mapwrite 0x03bf02 thru 0x03ffff (0x40fe bytes) 000671 1763402780.446080 write 0x02cc79 thru 0x03b7af (0xeb37 bytes) 000675 1763402780.468399 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x717321e2 (stale) Mirror 18: 0x717321e2 (stale) Mirror 20: 0x717321e2 (stale) Mirror 21: 0x717321e2 (stale) Mirror 22: 0x717321e2 (stale) Mirror 23: 0x717321e2 (stale) Mirror 24: 0x717321e2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xee98231 Mirror 25: 0xee98231 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000676 1763402781.248752 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x717321e2 (stale) Mirror 20: 0x717321e2 (stale) Mirror 21: 0x717321e2 (stale) Mirror 22: 0x717321e2 (stale) Mirror 23: 0x717321e2 (stale) Mirror 24: 0x717321e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x717321e2 (stale) Mirror 20: 0x717321e2 (stale) Mirror 21: 0x717321e2 (stale) Mirror 22: 0x717321e2 (stale) Mirror 23: 0x717321e2 (stale) Mirror 24: 0x717321e2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xee98231 Mirror 25: 0xee98231 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000677 1763402781.996554 write 0x007d26 thru 0x00a0de (0x23b9 bytes) truncating to largest ever: 0x3ff3a 000679 1763402782.100963 trunc from 0x040000 to 0x03ff39 (0xc6 bytes) 000680 1763402782.158773 write 0x01e366 thru 0x027ae2 (0x977d bytes) 000681 1763402782.173588 mapwrite 0x029189 thru 0x02ae94 (0x1d0c bytes) 000683 1763402782.258324 MIRROR_RESYNC 8 mirrors 000684 1763402783.261737 mapread 0x0103c9 thru 0x010893 (0x4cb bytes) 000687 1763402783.275211 MIRROR_EXTEND to 9 mirrors 000688 1763402783.935152 mapread 0x01eb0e thru 0x02787a (0x8d6d bytes) 000689 1763402783.960030 trunc from 0x03ff3a to 0x001ffd (0x3df3c bytes) 000691 1763402784.073007 mapread 0x00167b thru 0x001ffd (0x983 bytes) 000692 1763402784.080250 trunc from 0x001ffe to 0x01c381 (0x1a384 bytes) 000693 1763402784.104235 write 0x013de2 thru 0x01e90c (0xab2b bytes) 000695 1763402784.113981 read 0x00217d thru 0x007dff (0x5c83 bytes) 000697 1763402784.122405 trunc from 0x01e90d to 0x000dc9 (0x1db43 bytes) 000698 1763402784.143056 read 0x00042b thru 0x000dc9 (0x99f bytes) 000699 1763402784.150985 write 0x009a3e thru 0x00c720 (0x2ce3 bytes) 000700 1763402784.152021 trunc from 0x00c721 to 0x037a79 (0x2b359 bytes) 000701 1763402784.186353 MIRROR_RESYNC 9 mirrors 000702 1763402785.544663 read 0x014062 thru 0x0210ed (0xd08c bytes) 000705 1763402785.558745 mapwrite 0x023193 thru 0x028172 (0x4fe0 bytes) 000708 1763402785.755022 write 0x02d746 thru 0x03be18 (0xe6d3 bytes) 000712 1763402785.768491 mapread 0x00110d thru 0x004b87 (0x3a7b bytes) 000714 1763402785.781299 write 0x026456 thru 0x032d87 (0xc932 bytes) 000715 1763402785.782501 write 0x03c68d thru 0x03ffff (0x3973 bytes) 000716 1763402785.792192 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xce14edb2 (stale) Mirror 20: 0xce14edb2 (stale) Mirror 21: 0xce14edb2 (stale) Mirror 22: 0xce14edb2 (stale) Mirror 23: 0xce14edb2 (stale) Mirror 24: 0xce14edb2 (stale) Mirror 26: 0xce14edb2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xce14edb2 (stale) Mirror 20: 0xce14edb2 (stale) Mirror 21: 0xce14edb2 (stale) Mirror 22: 0xce14edb2 (stale) Mirror 23: 0xce14edb2 (stale) Mirror 24: 0xce14edb2 (stale) Mirror 26: 0xce14edb2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000717 1763402786.235947 read 0x02bf7a thru 0x035058 (0x90df bytes) 000718 1763402786.245547 read 0x036249 thru 0x03d902 (0x76ba bytes) 000719 1763402786.260984 mapwrite 0x00e69a thru 0x0187e3 (0xa14a bytes) 000720 1763402786.321324 write 0x020646 thru 0x0273d4 (0x6d8f bytes) 000722 1763402786.338970 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xce14edb2 (stale) Mirror 20: 0xce14edb2 (stale) Mirror 21: 0xce14edb2 (stale) Mirror 22: 0xce14edb2 (stale) Mirror 23: 0xce14edb2 (stale) Mirror 24: 0xce14edb2 (stale) Mirror 26: 0xce14edb2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb14bfd25 Mirror 27: 0xb14bfd25 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000723 1763402787.072253 trunc from 0x040000 to 0x012b09 (0x2d4f6 bytes) 000725 1763402787.233119 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x12b0a): Mirror 14: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12b0a): Mirror 14: 0xeb0fc830 (stale) Mirror 20: 0xeb0fc830 (stale) Mirror 21: 0xeb0fc830 (stale) Mirror 22: 0xeb0fc830 (stale) Mirror 23: 0xeb0fc830 (stale) Mirror 24: 0xeb0fc830 (stale) Mirror 26: 0xeb0fc830 (stale) Mirror 27: 0xc8809f04 (stale) CRC-32 checksum value for chunk [0, 0x12b0a): Mirror 1: 0xc8809f04 Mirror 28: 0xc8809f04 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12b0a: skipped 000727 1763402788.018680 trunc from 0x012b0a to 0x01ed3d (0xc234 bytes) 000728 1763402788.100928 read 0x0119f8 thru 0x0141ae (0x27b7 bytes) 000731 1763402788.108356 trunc from 0x01ed3e to 0x003162 (0x1bbdb bytes) 000732 1763402788.135347 read 0x0003bd thru 0x003162 (0x2da6 bytes) 000734 1763402788.141951 mapread 0x002aed thru 0x003162 (0x676 bytes) 000735 1763402788.142752 mapwrite 0x002495 thru 0x00c898 (0xa404 bytes) 000736 1763402788.248098 mapwrite 0x032d61 thru 0x03dc27 (0xaec7 bytes) 000738 1763402788.350275 trunc from 0x03dc28 to 0x0217e0 (0x1c447 bytes) 000740 1763402788.366529 write 0x010892 thru 0x01c9ce (0xc13d bytes) 000742 1763402788.368165 mapread 0x0006b1 thru 0x00691a (0x626a bytes) 000743 1763402788.368937 mapread 0x00e033 thru 0x00e481 (0x44f bytes) 000745 1763402788.375744 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x217e1): Mirror 14: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x217e1): Mirror 14: 0x3f10df8b (stale) Mirror 20: 0x3f10df8b (stale) Mirror 21: 0x3f10df8b (stale) Mirror 22: 0x3f10df8b (stale) Mirror 23: 0x3f10df8b (stale) Mirror 24: 0x3f10df8b (stale) Mirror 26: 0x3f10df8b (stale) Mirror 27: 0x161a2a30 (stale) Mirror 28: 0xbd467571 (stale) CRC-32 checksum value for chunk [0, 0x217e1): Mirror 1: 0x28c55bae Mirror 29: 0x28c55bae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x217e1: skipped 000746 1763402789.240661 trunc from 0x0217e1 to 0x029f20 (0x8740 bytes) 000747 1763402789.352971 mapwrite 0x022cbf thru 0x02bfca (0x930c bytes) 000748 1763402789.522417 write 0x020963 thru 0x023196 (0x2834 bytes) 000749 1763402789.523398 write 0x022044 thru 0x027963 (0x5920 bytes) 000751 1763402789.524713 mapread 0x025a47 thru 0x02bfca (0x6584 bytes) 000752 1763402789.527522 mapread 0x005c14 thru 0x00dc9d (0x808a bytes) 000754 1763402789.545671 read 0x022d6f thru 0x028726 (0x59b8 bytes) 000757 1763402789.547174 write 0x02f211 thru 0x03ba43 (0xc833 bytes) 000762 1763402789.558657 MIRROR_RESYNC 11 mirrors 000763 1763402791.235110 write 0x015566 thru 0x01f91e (0xa3b9 bytes) 000764 1763402791.329783 mapread 0x027d4f thru 0x02eae2 (0x6d94 bytes) 000765 1763402791.374959 mapread 0x030f91 thru 0x03b4e3 (0xa553 bytes) 000766 1763402791.390468 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ba44): Mirror 14: 0x30c44781 (stale) Mirror 20: 0x30c44781 (stale) Mirror 21: 0x30c44781 (stale) Mirror 22: 0x30c44781 (stale) Mirror 23: 0x30c44781 (stale) Mirror 24: 0x30c44781 (stale) Mirror 27: 0x30c44781 (stale) Mirror 28: 0x30c44781 (stale) Mirror 29: 0x30c44781 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba44): Mirror 14: 0x30c44781 (stale) Mirror 20: 0x30c44781 (stale) Mirror 21: 0x30c44781 (stale) Mirror 22: 0x30c44781 (stale) Mirror 23: 0x30c44781 (stale) Mirror 24: 0x30c44781 (stale) Mirror 27: 0x30c44781 (stale) Mirror 28: 0x30c44781 (stale) Mirror 29: 0x30c44781 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba44: skipped 000768 1763402791.926009 mapread 0x007ebd thru 0x013da9 (0xbeed bytes) 000769 1763402791.963771 trunc from 0x03ba44 to 0x015346 (0x266fd bytes) 000771 1763402791.982839 mapwrite 0x00cfbb thru 0x019557 (0xc59d bytes) 000776 1763402792.055156 trunc from 0x019558 to 0x0134e6 (0x6071 bytes) 000778 1763402792.082732 read 0x00657d thru 0x00ed0c (0x8790 bytes) 000779 1763402792.083363 trunc from 0x0134e7 to 0x01849a (0x4fb4 bytes) 000780 1763402792.105055 read 0x015dee thru 0x016c75 (0xe88 bytes) 000781 1763402792.112147 mapwrite 0x038144 thru 0x03a20c (0x20c9 bytes) 000782 1763402792.188475 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a20d): Mirror 14: 0x3403cd82 (stale) Mirror 20: 0x3403cd82 (stale) Mirror 21: 0x3403cd82 (stale) Mirror 22: 0x3403cd82 (stale) Mirror 23: 0x3403cd82 (stale) Mirror 24: 0x3403cd82 (stale) Mirror 27: 0x3403cd82 (stale) Mirror 29: 0x3403cd82 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a20d): Mirror 14: 0x3403cd82 (stale) Mirror 20: 0x3403cd82 (stale) Mirror 21: 0x3403cd82 (stale) Mirror 22: 0x3403cd82 (stale) Mirror 23: 0x3403cd82 (stale) Mirror 24: 0x3403cd82 (stale) Mirror 27: 0x3403cd82 (stale) Mirror 29: 0x3403cd82 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a20d: skipped 000784 1763402792.668105 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a20d): Mirror 20: 0x3403cd82 (stale) Mirror 21: 0x3403cd82 (stale) Mirror 22: 0x3403cd82 (stale) Mirror 23: 0x3403cd82 (stale) Mirror 24: 0x3403cd82 (stale) Mirror 27: 0x3403cd82 (stale) Mirror 29: 0x3403cd82 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a20d): Mirror 20: 0x3403cd82 (stale) Mirror 21: 0x3403cd82 (stale) Mirror 22: 0x3403cd82 (stale) Mirror 23: 0x3403cd82 (stale) Mirror 24: 0x3403cd82 (stale) Mirror 27: 0x3403cd82 (stale) Mirror 29: 0x3403cd82 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a20d: skipped 000785 1763402793.107708 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3a20d): Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a20d): Mirror 20: 0x3403cd82 (stale) Mirror 21: 0x3403cd82 (stale) Mirror 22: 0x3403cd82 (stale) Mirror 23: 0x3403cd82 (stale) Mirror 24: 0x3403cd82 (stale) Mirror 27: 0x3403cd82 (stale) Mirror 29: 0x3403cd82 (stale) CRC-32 checksum value for chunk [0, 0x3a20d): Mirror 1: 0x4b15c08a Mirror 30: 0x4b15c08a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a20d: skipped 000789 1763402794.052651 trunc from 0x03a20d to 0x035ed5 (0x4337 bytes) 000790 1763402794.153053 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35ed6): Mirror 20: 0x3121a058 (stale) Mirror 22: 0x3121a058 (stale) Mirror 23: 0x3121a058 (stale) Mirror 24: 0x3121a058 (stale) Mirror 27: 0x3121a058 (stale) Mirror 29: 0x3121a058 (stale) Mirror 30: 0x58ce52db (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35ed6): Mirror 20: 0x3121a058 (stale) Mirror 22: 0x3121a058 (stale) Mirror 23: 0x3121a058 (stale) Mirror 24: 0x3121a058 (stale) Mirror 27: 0x3121a058 (stale) Mirror 29: 0x3121a058 (stale) Mirror 30: 0x58ce52db (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35ed6: skipped 000792 1763402794.605771 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35ed6): Mirror 20: 0x3121a058 (stale) Mirror 22: 0x3121a058 (stale) Mirror 23: 0x3121a058 (stale) Mirror 24: 0x3121a058 (stale) Mirror 27: 0x3121a058 (stale) Mirror 30: 0x58ce52db (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35ed6): Mirror 20: 0x3121a058 (stale) Mirror 22: 0x3121a058 (stale) Mirror 23: 0x3121a058 (stale) Mirror 24: 0x3121a058 (stale) Mirror 27: 0x3121a058 (stale) Mirror 30: 0x58ce52db (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35ed6: skipped 000793 1763402794.967993 read 0x029453 thru 0x02b36f (0x1f1d bytes) 000794 1763402794.978746 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35ed6): Mirror 20: 0x3121a058 (stale) Mirror 22: 0x3121a058 (stale) Mirror 24: 0x3121a058 (stale) Mirror 27: 0x3121a058 (stale) Mirror 30: 0x58ce52db (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35ed6): Mirror 20: 0x3121a058 (stale) Mirror 22: 0x3121a058 (stale) Mirror 24: 0x3121a058 (stale) Mirror 27: 0x3121a058 (stale) Mirror 30: 0x58ce52db (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35ed6: skipped 000795 1763402795.530012 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x35ed6): Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35ed6): Mirror 20: 0x3121a058 (stale) Mirror 22: 0x3121a058 (stale) Mirror 24: 0x3121a058 (stale) Mirror 27: 0x3121a058 (stale) Mirror 30: 0x58ce52db (stale) CRC-32 checksum value for chunk [0, 0x35ed6): Mirror 1: 0x58ce52db Mirror 31: 0x58ce52db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35ed6: skipped 000797 1763402796.273572 mapwrite 0x01fc70 thru 0x028073 (0x8404 bytes) 000799 1763402796.520196 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35ed6): Mirror 20: 0x3121a058 (stale) Mirror 22: 0x3121a058 (stale) Mirror 24: 0x3121a058 (stale) Mirror 27: 0x3121a058 (stale) Mirror 31: 0x58ce52db (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35ed6): Mirror 20: 0x3121a058 (stale) Mirror 22: 0x3121a058 (stale) Mirror 24: 0x3121a058 (stale) Mirror 27: 0x3121a058 (stale) Mirror 31: 0x58ce52db (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35ed6: skipped 000801 1763402796.994718 write 0x00854e thru 0x010574 (0x8027 bytes) 000802 1763402797.008084 write 0x0114a0 thru 0x01b79f (0xa300 bytes) 000804 1763402797.024628 write 0x016d49 thru 0x01e9f9 (0x7cb1 bytes) 000806 1763402797.033153 write 0x028e0f thru 0x02bf4c (0x313e bytes) 000807 1763402797.042613 mapwrite 0x03ed89 thru 0x03ffff (0x1277 bytes) 000808 1763402797.137619 write 0x01b776 thru 0x01cabe (0x1349 bytes) 000812 1763402797.147058 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x1587890a (stale) Mirror 22: 0x1587890a (stale) Mirror 24: 0x1587890a (stale) Mirror 27: 0x1587890a (stale) Mirror 31: 0x4ed71568 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x667083c0 Mirror 32: 0x667083c0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000813 1763402798.011936 mapread 0x007a65 thru 0x00bbb5 (0x4151 bytes) 000814 1763402798.042579 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x1587890a (stale) Mirror 22: 0x1587890a (stale) Mirror 24: 0x1587890a (stale) Mirror 27: 0x1587890a (stale) Mirror 31: 0x4ed71568 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x667083c0 Mirror 32: 0x667083c0 Mirror 33: 0x667083c0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000817 1763402798.757696 trunc from 0x040000 to 0x001ce9 (0x3e316 bytes) 000819 1763402798.871841 read 0x0005c8 thru 0x001ce9 (0x1722 bytes) 000820 1763402798.881795 trunc from 0x001cea to 0x00dd61 (0xc078 bytes) 000821 1763402798.911691 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xdd62): Mirror 20: 0xf36f5409 (stale) Mirror 22: 0xf36f5409 (stale) Mirror 27: 0xf36f5409 (stale) Mirror 31: 0xe5be3a3 (stale) Mirror 32: 0xf92e38ed (stale) Mirror 33: 0xf92e38ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdd62): Mirror 20: 0xf36f5409 (stale) Mirror 22: 0xf36f5409 (stale) Mirror 27: 0xf36f5409 (stale) Mirror 31: 0xe5be3a3 (stale) Mirror 32: 0xf92e38ed (stale) Mirror 33: 0xf92e38ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdd62: skipped 000823 1763402799.251303 trunc from 0x00dd62 to 0x0300ac (0x2234b bytes) 000825 1763402799.270777 write 0x029e4f thru 0x03481d (0xa9cf bytes) 000827 1763402799.280695 mapread 0x014fb5 thru 0x02393b (0xe987 bytes) 000829 1763402799.302638 trunc from 0x03481e to 0x004641 (0x301dc bytes) 000830 1763402799.321767 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x4642): Mirror 22: 0xa3c937d9 (stale) Mirror 27: 0xa3c937d9 (stale) Mirror 31: 0xa3c937d9 (stale) Mirror 32: 0xa3c937d9 (stale) Mirror 33: 0xa3c937d9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4642): Mirror 22: 0xa3c937d9 (stale) Mirror 27: 0xa3c937d9 (stale) Mirror 31: 0xa3c937d9 (stale) Mirror 32: 0xa3c937d9 (stale) Mirror 33: 0xa3c937d9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4642: skipped 000831 1763402799.666011 mapread 0x0014f1 thru 0x004641 (0x3151 bytes) 000832 1763402799.687920 mapwrite 0x02b5f4 thru 0x038003 (0xca10 bytes) 000835 1763402799.778048 read 0x0328a9 thru 0x033cb1 (0x1409 bytes) 000836 1763402799.783943 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x38004): Mirror 22: (stale - read error) Mirror 27: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38004): Mirror 22: 0xc1c4596e (stale) Mirror 27: 0xc1c4596e (stale) Mirror 31: 0x30991c48 (stale) Mirror 32: 0x73d1bdef (stale) Mirror 33: 0x73d1bdef (stale) CRC-32 checksum value for chunk [0, 0x38004): Mirror 1: 0xb9d17b58 Mirror 34: 0xb9d17b58 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38004: skipped 000837 1763402800.759300 write 0x010f0d thru 0x016fa5 (0x6099 bytes) 000838 1763402800.857882 read 0x02a612 thru 0x032ff6 (0x89e5 bytes) 000840 1763402800.866706 read 0x02ea4a thru 0x034b82 (0x6139 bytes) 000841 1763402800.877443 read 0x00e76e thru 0x016057 (0x78ea bytes) 000842 1763402800.882343 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x38004): Mirror 22: (stale - read error) Mirror 27: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38004): Mirror 22: 0xc1c4596e (stale) Mirror 27: 0xc1c4596e (stale) Mirror 31: 0x30991c48 (stale) Mirror 32: 0x73d1bdef (stale) Mirror 33: 0x73d1bdef (stale) Mirror 34: 0xb9d17b58 (stale) CRC-32 checksum value for chunk [0, 0x38004): Mirror 1: 0x65cd6a38 Mirror 35: 0x65cd6a38 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38004: skipped 000843 1763402801.785963 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x38004): Mirror 22: (stale - read error) Mirror 27: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38004): Mirror 22: 0xc1c4596e (stale) Mirror 27: 0xc1c4596e (stale) Mirror 31: 0x30991c48 (stale) Mirror 32: 0x73d1bdef (stale) Mirror 33: 0x73d1bdef (stale) Mirror 34: 0xb9d17b58 (stale) CRC-32 checksum value for chunk [0, 0x38004): Mirror 1: 0x65cd6a38 Mirror 35: 0x65cd6a38 Mirror 36: 0x65cd6a38 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38004: skipped 000844 1763402802.515594 trunc from 0x038004 to 0x01478f (0x23874 bytes) 000846 1763402802.651746 mapwrite 0x02ae3e thru 0x03981e (0xe9e1 bytes) 000847 1763402802.753838 mapread 0x0166ab thru 0x017b1e (0x1474 bytes) 000848 1763402802.765218 write 0x00a744 thru 0x0132f9 (0x8bb6 bytes) 000853 1763402802.775349 read 0x038b65 thru 0x03981e (0xcba bytes) 000855 1763402802.783112 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3981f): Mirror 22: 0x8abb77ee (stale) Mirror 27: 0x8abb77ee (stale) Mirror 31: 0xa97de849 (stale) Mirror 33: 0xdf7498bb (stale) Mirror 34: 0x565ce31f (stale) Mirror 35: 0xb27ae18 (stale) Mirror 36: 0xb27ae18 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3981f): Mirror 22: 0x8abb77ee (stale) Mirror 27: 0x8abb77ee (stale) Mirror 31: 0xa97de849 (stale) Mirror 33: 0xdf7498bb (stale) Mirror 34: 0x565ce31f (stale) Mirror 35: 0xb27ae18 (stale) Mirror 36: 0xb27ae18 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3981f: skipped 000858 1763402803.233872 write 0x00816d thru 0x00e5f0 (0x6484 bytes) 000859 1763402803.254697 read 0x004d6b thru 0x00db84 (0x8e1a bytes) 000862 1763402803.262146 mapread 0x031359 thru 0x0338ac (0x2554 bytes) 000863 1763402803.282622 write 0x01f690 thru 0x028798 (0x9109 bytes) 000865 1763402803.305849 read 0x016806 thru 0x01af19 (0x4714 bytes) 000867 1763402803.322281 mapread 0x03242b thru 0x03981e (0x73f4 bytes) 000868 1763402803.323931 mapwrite 0x029160 thru 0x035936 (0xc7d7 bytes) 000869 1763402803.384393 mapwrite 0x019cef thru 0x01be84 (0x2196 bytes) 000870 1763402803.427760 write 0x031303 thru 0x037895 (0x6593 bytes) 000871 1763402803.428735 write 0x02f80b thru 0x02fa2b (0x221 bytes) 000874 1763402803.430251 trunc from 0x03981f to 0x0393ce (0x450 bytes) 000877 1763402803.451101 mapwrite 0x007a94 thru 0x00acec (0x3259 bytes) 000880 1763402803.513351 mapread 0x03316c thru 0x0393ce (0x6263 bytes) 000883 1763402803.516544 write 0x02a73d thru 0x02c1bd (0x1a81 bytes) 000886 1763402803.521135 write 0x0142f6 thru 0x02272f (0xe43a bytes) 000888 1763402803.525186 trunc from 0x0393cf to 0x0228d5 (0x16af9 bytes) 000889 1763402803.554057 mapwrite 0x0187a6 thru 0x023e58 (0xb6b3 bytes) 000890 1763402803.654543 trunc from 0x023e59 to 0x023d77 (0xe1 bytes) 000891 1763402803.681400 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23d78): Mirror 27: 0xcda47d56 (stale) Mirror 31: 0xc73f0a50 (stale) Mirror 33: 0x2b3ca59e (stale) Mirror 34: 0x23fdca00 (stale) Mirror 35: 0xf10f4a57 (stale) Mirror 36: 0xf10f4a57 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23d78): Mirror 27: 0xcda47d56 (stale) Mirror 31: 0xc73f0a50 (stale) Mirror 33: 0x2b3ca59e (stale) Mirror 34: 0x23fdca00 (stale) Mirror 35: 0xf10f4a57 (stale) Mirror 36: 0xf10f4a57 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23d78: skipped 000893 1763402804.055516 read 0x003565 thru 0x0119ef (0xe48b bytes) 000895 1763402804.065783 read 0x014839 thru 0x01e938 (0xa100 bytes) 000896 1763402804.077627 write 0x0369a5 thru 0x03c076 (0x56d2 bytes) 000897 1763402804.083241 MIRROR_RESYNC 7 mirrors 000898 1763402805.296860 mapread 0x036c9b thru 0x03c076 (0x53dc bytes) 000900 1763402805.322224 trunc from 0x03c077 to 0x008417 (0x33c5f bytes) 000904 1763402805.458646 write 0x02c5a1 thru 0x034b96 (0x85f6 bytes) 000905 1763402805.461132 write 0x03953b thru 0x03ffff (0x6ac5 bytes) 000906 1763402805.463409 trunc from 0x040000 to 0x035e28 (0xa1d7 bytes) 000908 1763402805.490129 mapread 0x021ec9 thru 0x02fe0e (0xdf46 bytes) 000909 1763402805.509898 read 0x005f60 thru 0x012e49 (0xceea bytes) 000910 1763402805.519964 mapread 0x018097 thru 0x01e89d (0x6807 bytes) 000911 1763402805.530391 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35e29): Mirror 31: 0x9e7184a3 (stale) Mirror 33: 0x9e7184a3 (stale) Mirror 34: 0x9e7184a3 (stale) Mirror 35: 0x9e7184a3 (stale) Mirror 36: 0x9e7184a3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35e29): Mirror 31: 0x9e7184a3 (stale) Mirror 33: 0x9e7184a3 (stale) Mirror 34: 0x9e7184a3 (stale) Mirror 35: 0x9e7184a3 (stale) Mirror 36: 0x9e7184a3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35e29: skipped 000912 1763402805.958643 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x35e29): Mirror 31: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35e29): Mirror 31: 0x9e7184a3 (stale) Mirror 33: 0x9e7184a3 (stale) Mirror 34: 0x9e7184a3 (stale) Mirror 35: 0x9e7184a3 (stale) Mirror 36: 0x9e7184a3 (stale) CRC-32 checksum value for chunk [0, 0x35e29): Mirror 1: 0xfc7e73cc Mirror 37: 0xfc7e73cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35e29: skipped 000914 1763402806.745859 read 0x00f88e thru 0x01f84d (0xffc0 bytes) 000916 1763402806.775166 mapread 0x01c348 thru 0x0204cc (0x4185 bytes) 000917 1763402806.802582 mapwrite 0x01772d thru 0x02162c (0x9f00 bytes) 000918 1763402806.982225 read 0x030bb8 thru 0x030ffd (0x446 bytes) 000920 1763402806.990911 read 0x014d5a thru 0x0209c7 (0xbc6e bytes) 000922 1763402806.993134 mapread 0x03177b thru 0x035e28 (0x46ae bytes) 000923 1763402806.993980 write 0x00b63d thru 0x019d4a (0xe70e bytes) 000925 1763402807.000386 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35e29): Mirror 31: 0x9e7184a3 (stale) Mirror 34: 0x9e7184a3 (stale) Mirror 35: 0x9e7184a3 (stale) Mirror 36: 0x9e7184a3 (stale) Mirror 37: 0xfc7e73cc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35e29): Mirror 31: 0x9e7184a3 (stale) Mirror 34: 0x9e7184a3 (stale) Mirror 35: 0x9e7184a3 (stale) Mirror 36: 0x9e7184a3 (stale) Mirror 37: 0xfc7e73cc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35e29: skipped 000926 1763402807.405857 read 0x004861 thru 0x00cd7e (0x851e bytes) 000928 1763402807.414003 mapread 0x01be18 thru 0x02a261 (0xe44a bytes) 000930 1763402807.439458 MIRROR_RESYNC 6 mirrors 000933 1763402808.487140 trunc from 0x035e29 to 0x009069 (0x2cdbf bytes) 000934 1763402808.558261 trunc from 0x00906a to 0x03bab1 (0x32a48 bytes) 000937 1763402808.578165 read 0x02c89a thru 0x02e07b (0x17e2 bytes) 000938 1763402808.585316 read 0x02542c thru 0x02fdec (0xa9c1 bytes) 000940 1763402808.596850 mapwrite 0x022b60 thru 0x023ab0 (0xf51 bytes) 000942 1763402808.636232 write 0x03eca9 thru 0x03ffff (0x1357 bytes) 000944 1763402808.638120 write 0x032bf0 thru 0x0330a6 (0x4b7 bytes) 000946 1763402808.640094 write 0x02f31f thru 0x036f70 (0x7c52 bytes) 000947 1763402808.643621 mapread 0x0334d8 thru 0x037829 (0x4352 bytes) 000949 1763402808.652744 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 31: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 31: 0x53c3f4 (stale) Mirror 34: 0x53c3f4 (stale) Mirror 35: 0x53c3f4 (stale) Mirror 36: 0x53c3f4 (stale) Mirror 37: 0x53c3f4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf19abb43 Mirror 38: 0xf19abb43 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000952 1763402809.616895 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 34: 0x53c3f4 (stale) Mirror 35: 0x53c3f4 (stale) Mirror 36: 0x53c3f4 (stale) Mirror 37: 0x53c3f4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 34: 0x53c3f4 (stale) Mirror 35: 0x53c3f4 (stale) Mirror 36: 0x53c3f4 (stale) Mirror 37: 0x53c3f4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf19abb43 Mirror 38: 0xf19abb43 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000954 1763402810.357074 trunc from 0x040000 to 0x01a939 (0x256c6 bytes) 000955 1763402810.460503 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1a93a): Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) Mirror 38: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a93a): Mirror 34: 0x59a20f54 (stale) Mirror 35: 0x59a20f54 (stale) Mirror 36: 0x59a20f54 (stale) Mirror 37: 0x59a20f54 (stale) Mirror 38: 0x4cf76a2b (stale) CRC-32 checksum value for chunk [0, 0x1a93a): Mirror 1: 0x4cf76a2b Mirror 39: 0x4cf76a2b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a93a: skipped 000956 1763402811.273244 mapread 0x0031c2 thru 0x005f0d (0x2d4c bytes) 000958 1763402811.313384 write 0x036707 thru 0x03fd9a (0x9694 bytes) 000959 1763402811.381181 mapread 0x031ffd thru 0x0362f8 (0x42fc bytes) 000960 1763402811.393761 mapread 0x023752 thru 0x032b9b (0xf44a bytes) 000961 1763402811.459936 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3fd9b): Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) Mirror 38: (stale - read error) Mirror 39: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd9b): Mirror 34: 0x4ac9f20c (stale) Mirror 35: 0x4ac9f20c (stale) Mirror 36: 0x4ac9f20c (stale) Mirror 37: 0x4ac9f20c (stale) Mirror 38: 0x18b225a6 (stale) Mirror 39: 0xf7ebc58c (stale) CRC-32 checksum value for chunk [0, 0x3fd9b): Mirror 1: 0x88c5448d Mirror 40: 0x88c5448d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd9b: skipped 000965 1763402812.435395 trunc from 0x03fd9b to 0x02c0cc (0x13cce bytes) 000966 1763402812.579664 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2c0cd): Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) Mirror 38: (stale - read error) Mirror 39: (stale - read error) Mirror 40: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c0cd): Mirror 34: 0x802d99c6 (stale) Mirror 35: 0x802d99c6 (stale) Mirror 36: 0x802d99c6 (stale) Mirror 37: 0x802d99c6 (stale) Mirror 38: 0x57352700 (stale) Mirror 39: 0xd058c869 (stale) Mirror 40: 0xd058c869 (stale) CRC-32 checksum value for chunk [0, 0x2c0cd): Mirror 1: 0xd058c869 Mirror 41: 0xd058c869 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c0cd: skipped 000967 1763402813.428128 trunc from 0x02c0cd to 0x019b3b (0x12591 bytes) 000970 1763402813.530672 write 0x010070 thru 0x0168a5 (0x6836 bytes) 000972 1763402813.547138 write 0x0141fa thru 0x01c884 (0x868b bytes) 000976 1763402813.561274 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1c885): Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) Mirror 38: (stale - read error) Mirror 39: (stale - read error) Mirror 40: (stale - read error) Mirror 41: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c885): Mirror 34: 0x6e48ed23 (stale) Mirror 35: 0x6e48ed23 (stale) Mirror 36: 0x6e48ed23 (stale) Mirror 37: 0x6e48ed23 (stale) Mirror 38: 0x44e4c47e (stale) Mirror 39: 0x44e4c47e (stale) Mirror 40: 0x44e4c47e (stale) Mirror 41: 0x44e4c47e (stale) CRC-32 checksum value for chunk [0, 0x1c885): Mirror 1: 0x468da356 Mirror 42: 0x468da356 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c885: skipped 000978 1763402814.698316 mapwrite 0x03a1aa thru 0x03ffff (0x5e56 bytes) 000979 1763402814.917375 trunc from 0x040000 to 0x0273ac (0x18c53 bytes) 000980 1763402814.949852 mapread 0x006036 thru 0x00e926 (0x88f1 bytes) 000981 1763402814.965771 read 0x01caa5 thru 0x023ab2 (0x700e bytes) 000982 1763402815.000958 read 0x023dcc thru 0x0273ac (0x35e1 bytes) 000983 1763402815.026151 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x273ad): Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) Mirror 38: (stale - read error) Mirror 39: (stale - read error) Mirror 40: (stale - read error) Mirror 41: (stale - read error) Mirror 42: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x273ad): Mirror 34: 0xfdd041df (stale) Mirror 35: 0xfdd041df (stale) Mirror 36: 0xfdd041df (stale) Mirror 37: 0xfdd041df (stale) Mirror 38: 0x26f80230 (stale) Mirror 39: 0x38017a41 (stale) Mirror 40: 0x38017a41 (stale) Mirror 41: 0x38017a41 (stale) Mirror 42: 0x359834a5 (stale) CRC-32 checksum value for chunk [0, 0x273ad): Mirror 1: 0x359834a5 Mirror 43: 0x359834a5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x273ad: skipped 000985 1763402815.903716 read 0x010781 thru 0x014986 (0x4206 bytes) 000987 1763402815.933074 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x273ad): Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) Mirror 38: (stale - read error) Mirror 39: (stale - read error) Mirror 40: (stale - read error) Mirror 41: (stale - read error) Mirror 42: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x273ad): Mirror 34: 0xfdd041df (stale) Mirror 35: 0xfdd041df (stale) Mirror 36: 0xfdd041df (stale) Mirror 37: 0xfdd041df (stale) Mirror 38: 0x26f80230 (stale) Mirror 39: 0x38017a41 (stale) Mirror 40: 0x38017a41 (stale) Mirror 41: 0x38017a41 (stale) Mirror 42: 0x359834a5 (stale) CRC-32 checksum value for chunk [0, 0x273ad): Mirror 1: 0x359834a5 Mirror 43: 0x359834a5 Mirror 44: 0x359834a5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x273ad: skipped 000988 1763402816.808152 write 0x026cab thru 0x03630c (0xf662 bytes) 000989 1763402816.905258 read 0x030db8 thru 0x03630c (0x5555 bytes) 000990 1763402816.907222 mapwrite 0x03ba14 thru 0x03ffff (0x45ec bytes) 000993 1763402817.019552 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) Mirror 38: (stale - read error) Mirror 39: (stale - read error) Mirror 40: (stale - read error) Mirror 41: (stale - read error) Mirror 42: (stale - read error) Mirror 43: (stale - read error) Mirror 44: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 34: 0x53c3f4 (stale) Mirror 35: 0x53c3f4 (stale) Mirror 36: 0x53c3f4 (stale) Mirror 37: 0x53c3f4 (stale) Mirror 38: 0xf19abb43 (stale) Mirror 39: 0xb1c34c3c (stale) Mirror 40: 0xd756733c (stale) Mirror 41: 0xb1c34c3c (stale) Mirror 42: 0x32b771e8 (stale) Mirror 43: 0x32b771e8 (stale) Mirror 44: 0x32b771e8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xff432429 Mirror 45: 0xff432429 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000994 1763402818.194028 MIRROR_RESYNC 13 mirrors 000997 1763402819.815932 mapwrite 0x01c784 thru 0x0214a0 (0x4d1d bytes) 000998 1763402820.043118 read 0x01c09e thru 0x028d9e (0xcd01 bytes) All operations completed A-OK! PASS 70a (115s) == sanity-flr test 71: check mirror extend parent fid ==== 13:07:04 (1763402824) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 200a: stress test ===================== 13:07:07 (1763402827) Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre2 Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6741312 .. Extending file size to 6737088 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8380480 .. Extending file size to 7621024 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8445056 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8259392 .. failed Extending file size to 8492320 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8375584 .. failed Extending file size to 8639520 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8889728 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8926720 .. Extending file size to 8984160 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 55151 55152 55153 55156 55157 done 192.168.204.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg452-client.virtnet /mnt/lustre2 (opts:) 192.168.204.152@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg452-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (78s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 13:08:25 (1763402905) Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre2 Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1875040 .. Extending file size to 5384320 .. done Extending file size to 3314464 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 4684640 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7983072 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8144096 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8316288 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8624384 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9001856 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9174976 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57273 57275 57276 57277 57278 done 192.168.204.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg452-client.virtnet /mnt/lustre2 (opts:) 192.168.204.152@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg452-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (74s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 13:09:40 (1763402980) Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4224 2204160 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4224 2204160 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0775568 s, 27.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.273568 s, 7.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x33935d3b (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg452-client.virtnet /mnt/lustre2 (opts:) PASS 200c (15s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 13:09:57 (1763402997) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.101144 s, 20.7 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 13:10:03 (1763403003) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x266:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x306:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x306:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x307:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x309:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 13:10:09 (1763403009) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.17045 s, 12.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x308:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (6s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 13:10:16 (1763403016) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.211741 s, 9.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30d:0x0] } PASS 204b (5s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 13:10:22 (1763403022) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.314533 s, 6.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30f:0x0] } PASS 204c (30s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 13:10:52 (1763403052) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg452-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762783 s, 13.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x311:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x310:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x311:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x310:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } PASS 204d (52s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 13:11:45 (1763403105) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg452-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.191419 s, 11.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x312:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x312:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (50s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 13:12:35 (1763403155) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg452-server mds-ost sync done. Creating new pool test_204f oleg452-server: Pool lustre.test_204f created Adding targets to pool oleg452-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg452-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.321781 s, 6.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x313:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x313:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg452-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg452-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg452-server: Pool lustre.test_204f destroyed PASS 204f (72s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 13:13:47 (1763403227) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 13:13:54 (1763403234) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 13:14:01 (1763403241) oleg452-server: Pool lustre.test_206 created Adding targets to pool oleg452-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg452-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg452-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg452-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg452-server: Pool lustre.test_206 destroyed PASS 206 (24s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 13:14:25 (1763403265) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114461 s, 9.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297877 s, 35.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xe95cdf4e Mirror 3: 0xe95cdf4e PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 13:14:35 (1763403275) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 13:14:39 (1763403279) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 13:14:43 (1763403283) Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.111712 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x24d0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1fb0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1fb0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1fb0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1840000 Read this (4 bytes): BBBB 192.168.204.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg452-client.virtnet /mnt/lustre2 (opts:) PASS 209a (32s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 13:15:15 (1763403315) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0257039 s, 319 kB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 13:15:20 (1763403320) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391761 s, 26.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x28b lmm_fid: [0x200000404:0x28b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x28b lmm_fid: [0x200000404:0x28b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x31d:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834571 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-17 13:15:22.000000000 -0500 Modify: 2025-11-17 13:15:22.000000000 -0500 Change: 2025-11-17 13:15:23.000000000 -0500 Birth: 2025-11-17 13:15:21.000000000 -0500 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 13:15:28 (1763403328) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 13:15:30 (1763403330) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.425411 s, 96.3 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0487978 s, 83.9 kB/s size after second write -rw-r--r-- 1 root root 4096 Nov 17 13:15 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x28d:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Nov 17 13:15 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 13:15:38 (1763403338) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 13:15:45 (1763403345) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (7s) == sanity-flr test complete, duration 1496 sec =========== 13:15:52 (1763403352) === sanity-flr: start cleanup 13:15:54 (1763403354) === === sanity-flr: finish cleanup 13:16:00 (1763403360) ===