-----============= acceptance-small: sanity-flr ============----- Mon Mar 2 01:51:18 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg316-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 01:51:36 (1772434296) === oleg316-client.virtnet: executing check_config_client /mnt/lustre oleg316-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg316-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8b604353b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b604353b800.idle_timeout=debug disable quota as required oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 01:52:11 (1772434331) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7571] == sanity-flr test 0a: lfs mirror create with -N option == 01:52:13 (1772434333) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:52:27 (1772434347) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:52:30 (1772434350) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:52:33 (1772434353) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:52:56 (1772434376) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:52:59 (1772434379) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 01:53:03 (1772434383) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8b604353b800.stats=clear osc.lustre-OST0001-osc-ffff8b604353b800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:53:10 (1772434390) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:53:21 (1772434401) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:53:29 (1772434409) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:53:33 (1772434413) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 01:53:42 (1772434422) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:53:50 (1772434430) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:53:55 (1772434435) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:54:03 (1772434443) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:54:14 (1772434454) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:54:21 (1772434461) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.539345 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849481 s, 12.3 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:54:31 (1772434471) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.41103 s, 33.2 MB/s PASS 22 (8s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:54:40 (1772434480) Stopping /mnt/lustre-ost1 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b604353b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b604353b800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg316-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8b604353b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b604353b800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg316-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (71s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:55:51 (1772434551) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.39871 s, 18.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b604353b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b604353b800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg316-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (35s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:56:26 (1772434586) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b604353b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b604353b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg316-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8b604353b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b604353b800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg316-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (66s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:57:32 (1772434652) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.97128 s, 13.2 MB/s -rw-r--r-- 1 root root 52428800 Mar 2 01:57 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.467s user 0m0.000s sys 0m0.173s Stopping /mnt/lustre-ost1 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b604353b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b604353b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.288s user 0m0.001s sys 0m0.182s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg316-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8b604353b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b604353b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.394s user 0m0.001s sys 0m0.191s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg316-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg316-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b604353b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b604353b800.ost_server_uuid in FULL state after 0 sec PASS 33b (89s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:59:01 (1772434741) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:59:05 (1772434745) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:59:09 (1772434749) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 01:59:12 (1772434752) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 01:59:20 (1772434760) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.651734 s, 19.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:59:33 (1772434773) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310234 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207163 s, 50.6 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:59:40 (1772434780) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:59:48 (1772434788) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000801776 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00145429 s, 704 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00463535 s, 221 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00210874 s, 486 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 01:59:59 (1772434799) create test files with size 15751471 .. 434d7b1f12b0adeb4584bd76d9e0016e - b7f4955f89998f79cbc2bcf05f62d9aa - 65890e812f61b33f24b1e9fa54e986c1 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg316-client.virtnet /mnt/lustre (opts:) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 02:00:22 (1772434822) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 02:00:37 (1772434837) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 02:00:42 (1772434842) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749124 s, 14.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475718 s, 22.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 02:00:50 (1772434850) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113345 s, 18.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.465673 s, 9.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.19002 s, 11.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 02:01:10 (1772434870) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 02:01:14 (1772434874) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 02:01:18 (1772434878) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375521 s, 27.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 02:01:27 (1772434887) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 02:01:30 (1772434890) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 02:01:38 (1772434898) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.230138 s, 45.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg316-client.virtnet /mnt/lustre (opts:) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 02:01:49 (1772434909) llite.lustre-ffff8b6059503000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.227138 s, 46.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg316-client.virtnet /mnt/lustre (opts:) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8b6049aa8800.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 02:02:03 (1772434923) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (35s) == sanity-flr test 46: Verify setstripe --copy option ==== 02:02:38 (1772434958) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 02:03:13 (1772434993) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 02:03:17 (1772434997) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.268225 s, 11.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.177083 s, 17.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xda18969e (stale) Mirror 2: 0xda18969e (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 8af494b37ffeb8632c6c4ca64e63eb7e - mirror 1 checksum: f4ceec1a186dd0353419990a3611eac4 - mirror 2 checksum: 8af494b37ffeb8632c6c4ca64e63eb7e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Mar 2 02:03:21 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Mar 2 02:03:21 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 8af494b37ffeb8632c6c4ca64e63eb7e - mirror 1 checksum: f4ceec1a186dd0353419990a3611eac4 - mirror 2 checksum: f4ceec1a186dd0353419990a3611eac4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Mar 2 02:03:21 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Mar 2 02:03:21 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 02:03:29 (1772435009) Failing mds1 on oleg316-server Stopping /mnt/lustre-mds1 (opts:) on oleg316-server 02:03:36 (1772435016) shut down facet: mds1 facet_host: oleg316-server facet_failover_host: oleg316-server Failover mds1 to oleg316-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-MDT0000 02:03:57 (1772435037) targets are mounted 02:03:57 (1772435037) facet_failover done oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462687 s, 22.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b:0x0] } PASS 50A (40s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 02:04:10 (1772435050) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 02:04:13 (1772435053) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 02:04:16 (1772435056) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0109325 s, 1.5 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 02:04:22 (1772435062) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 02:04:25 (1772435065) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 02:04:29 (1772435069) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 02:04:54 (1772435094) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (32s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 02:05:27 (1772435127) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 02:05:51 (1772435151) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 02:06:15 (1772435175) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 02:06:23 (1772435183) mirror create pid 52309 mirror split pid 52310 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52309 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52310 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 02:07:31 (1772435251) Seed set to 5253 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x2e1ef 000001 1772435253.780629 trunc from 00000000 to 0x02e1ee (0x2e1ef bytes) 000003 1772435253.817087 MIRROR_EXTEND to 2 mirrors 000007 1772435254.638070 read 0x01ed7a thru 0x025839 (0x6ac0 bytes) 000008 1772435254.663544 trunc from 0x02e1ef to 0x00b290 (0x22f5e bytes) truncating to largest ever: 0x31fea 000010 1772435254.775033 trunc from 0x00b291 to 0x031fe9 (0x26d59 bytes) 000011 1772435254.805031 mapread 0x00f7a7 thru 0x012bd6 (0x3430 bytes) 000012 1772435254.846027 read 0x00a9e1 thru 0x017c83 (0xd2a3 bytes) 000015 1772435254.848049 trunc from 0x031fea to 0x00e377 (0x23c72 bytes) 000016 1772435254.882727 mapwrite 0x03a603 thru 0x03ffff (0x59fd bytes) 000020 1772435255.055482 write 0x02f897 thru 0x038567 (0x8cd1 bytes) 000025 1772435255.061435 mapwrite 0x031c96 thru 0x03ffff (0xe36a bytes) 000026 1772435255.124790 write 0x01a72f thru 0x01f6e1 (0x4fb3 bytes) 000027 1772435255.126024 mapwrite 0x004165 thru 0x00e0e1 (0x9f7d bytes) 000028 1772435255.267629 read 0x00292e thru 0x00d4ec (0xabbf bytes) 000029 1772435255.269084 mapread 0x0121cf thru 0x01cfa8 (0xadda bytes) 000031 1772435255.285437 trunc from 0x040000 to 0x01904d (0x26fb2 bytes) 000033 1772435255.327979 write 0x001215 thru 0x00fcdf (0xeacb bytes) 000035 1772435255.334551 mapread 0x00ee7b thru 0x01904d (0xa1d3 bytes) 000037 1772435255.340706 mapread 0x012724 thru 0x01904d (0x692a bytes) truncating to largest ever: 0x32251 000038 1772435255.347366 trunc from 0x01904e to 0x032250 (0x19203 bytes) 000040 1772435255.375254 trunc from 0x032251 to 0x008674 (0x29bdc bytes) 000041 1772435255.409513 mapread 0x004a8b thru 0x0066a5 (0x1c1b bytes) 000044 1772435255.416542 mapwrite 0x01de63 thru 0x02a8be (0xca5c bytes) 000047 1772435255.631391 write 0x01637e thru 0x025c47 (0xf8ca bytes) 000048 1772435255.633103 mapwrite 0x02a3f7 thru 0x02f951 (0x555b bytes) 000050 1772435255.826076 write 0x032378 thru 0x03ffff (0xdc88 bytes) 000052 1772435255.840477 mapwrite 0x02bc0e thru 0x038ef0 (0xd2e3 bytes) 000056 1772435255.945432 trunc from 0x040000 to 0x029f5a (0x160a5 bytes) 000058 1772435255.969429 mapwrite 0x01a190 thru 0x021f4d (0x7dbe bytes) 000060 1772435256.059515 write 0x01698f thru 0x017e18 (0x148a bytes) 000061 1772435256.061012 read 0x023f03 thru 0x029f5a (0x6058 bytes) 000063 1772435256.063975 read 0x007573 thru 0x00d625 (0x60b3 bytes) 000064 1772435256.065176 write 0x03e66d thru 0x03ffff (0x1993 bytes) 000066 1772435256.076805 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xae77710d Mirror 3: 0xae77710d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000067 1772435257.081154 write 0x0207fc thru 0x0306f4 (0xfef9 bytes) 000068 1772435257.260123 write 0x010b14 thru 0x01891c (0x7e09 bytes) 000073 1772435257.279006 mapwrite 0x03f0a1 thru 0x03ffff (0xf5f bytes) 000076 1772435257.395069 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000077 1772435257.685633 write 0x0242fd thru 0x02a7f1 (0x64f5 bytes) truncating to largest ever: 0x38ba2 000078 1772435257.693530 trunc from 0x040000 to 0x038ba1 (0x745e bytes) 000079 1772435257.730374 read 0x0204cb thru 0x02634c (0x5e82 bytes) 000081 1772435257.740918 read 0x01b906 thru 0x025657 (0x9d52 bytes) 000082 1772435257.750904 mapread 0x02d5d9 thru 0x032e64 (0x588c bytes) 000083 1772435257.752064 mapread 0x0251fe thru 0x02b50a (0x630d bytes) 000084 1772435257.754186 mapwrite 0x01c561 thru 0x021fe0 (0x5a80 bytes) 000085 1772435257.804999 mapwrite 0x020e0f thru 0x022777 (0x1969 bytes) 000086 1772435257.863598 read 0x02fbbe thru 0x03869a (0x8add bytes) 000087 1772435257.864347 trunc from 0x038ba2 to 0x01ab7e (0x1e023 bytes) 000089 1772435257.887553 mapwrite 0x02c437 thru 0x03b8c0 (0xf48a bytes) 000091 1772435257.992079 trunc from 0x03b8c1 to 0x0344cd (0x73f3 bytes) 000092 1772435258.016184 mapwrite 0x028c87 thru 0x02ab14 (0x1e8e bytes) 000093 1772435258.064923 read 0x01edd8 thru 0x02b09d (0xc2c6 bytes) 000094 1772435258.065729 trunc from 0x0344ce to 0x034616 (0x149 bytes) 000095 1772435258.085032 read 0x01579f thru 0x016f81 (0x17e3 bytes) 000097 1772435258.086305 trunc from 0x034617 to 0x002f01 (0x31715 bytes) 000098 1772435258.108132 read 0x002dfa thru 0x002f01 (0x108 bytes) 000099 1772435258.109095 mapwrite 0x02bd34 thru 0x037b1c (0xbde9 bytes) 000100 1772435258.191979 mapwrite 0x01f676 thru 0x02ca7c (0xd407 bytes) 000101 1772435258.260908 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) 000102 1772435258.388347 write 0x003d63 thru 0x013bc2 (0xfe60 bytes) truncating to largest ever: 0x3c84f 000103 1772435258.403499 trunc from 0x037b1d to 0x03c84e (0x4d32 bytes) 000104 1772435258.426655 trunc from 0x03c84f to 0x02bddf (0x10a6f bytes) 000107 1772435258.449848 write 0x014ddc thru 0x021804 (0xca29 bytes) 000108 1772435258.466978 write 0x01b574 thru 0x023e0b (0x8898 bytes) 000109 1772435258.484771 MIRROR_EXTEND to 2 mirrors 000112 1772435259.570253 mapread 0x01fc80 thru 0x022e2d (0x31ae bytes) 000115 1772435259.663191 write 0x00bece thru 0x0187ee (0xc921 bytes) 000117 1772435259.863758 write 0x01ffa9 thru 0x026ecf (0x6f27 bytes) 000118 1772435259.899373 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) 000120 1772435260.140775 mapwrite 0x0238ea thru 0x032428 (0xeb3f bytes) 000121 1772435260.287395 mapread 0x00eaf3 thru 0x014368 (0x5876 bytes) 000123 1772435260.307268 read 0x00e39a thru 0x01e055 (0xfcbc bytes) 000129 1772435260.320400 mapread 0x0083d8 thru 0x016b4f (0xe778 bytes) 000132 1772435260.324112 write 0x02d11d thru 0x037bd2 (0xaab6 bytes) 000133 1772435260.326522 mapwrite 0x03ce92 thru 0x03d91c (0xa8b bytes) 000134 1772435260.415372 read 0x006699 thru 0x00d6d2 (0x703a bytes) 000138 1772435260.431365 read 0x001b5d thru 0x00c70c (0xabb0 bytes) 000139 1772435260.432361 mapwrite 0x00a743 thru 0x00d1f9 (0x2ab7 bytes) 000140 1772435260.485391 trunc from 0x03d91d to 0x027605 (0x16317 bytes) 000141 1772435260.515315 MIRROR_EXTEND to 2 mirrors 000142 1772435261.091453 write 0x03cdbc thru 0x03ffff (0x3244 bytes) 000143 1772435261.232685 mapwrite 0x03ba2f thru 0x03ffff (0x45d1 bytes) 000144 1772435261.434765 mapwrite 0x03b2d5 thru 0x03ffff (0x4d2b bytes) 000145 1772435261.601846 read 0x02d8c2 thru 0x02dbeb (0x32a bytes) 000147 1772435261.614133 mapread 0x01aac6 thru 0x02543c (0xa977 bytes) 000148 1772435261.674850 read 0x000ac8 thru 0x00276f (0x1ca8 bytes) 000150 1772435261.686100 mapwrite 0x037099 thru 0x03ffff (0x8f67 bytes) 000151 1772435261.809941 read 0x003e43 thru 0x00f08d (0xb24b bytes) 000152 1772435261.810994 mapwrite 0x03f9cd thru 0x03ffff (0x633 bytes) 000154 1772435261.928905 write 0x006226 thru 0x012a82 (0xc85d bytes) 000155 1772435261.930417 write 0x01d2ed thru 0x02183f (0x4553 bytes) 000156 1772435261.941470 mapwrite 0x02c2af thru 0x035a6b (0x97bd bytes) 000158 1772435262.050908 trunc from 0x040000 to 0x02420f (0x1bdf0 bytes) 000161 1772435262.151550 MIRROR_RESYNC 2 mirrors 000162 1772435262.800717 write 0x02e72c thru 0x03935a (0xac2f bytes) 000164 1772435262.863031 write 0x00fb38 thru 0x016765 (0x6c2e bytes) 000165 1772435262.874498 trunc from 0x03935b to 0x0232d0 (0x1608a bytes) 000169 1772435262.904466 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) 000172 1772435263.052510 write 0x017bdf thru 0x0277a9 (0xfbcb bytes) 000173 1772435263.057830 write 0x038e6f thru 0x03ffff (0x7191 bytes) 000175 1772435263.059320 trunc from 0x040000 to 0x02a385 (0x15c7a bytes) 000176 1772435263.081718 mapread 0x0146a4 thru 0x023c13 (0xf570 bytes) 000179 1772435263.108328 write 0x03ed31 thru 0x03ffff (0x12cf bytes) 000180 1772435263.109824 mapread 0x03d38b thru 0x03ffff (0x2c75 bytes) 000181 1772435263.120928 trunc from 0x040000 to 0x01f830 (0x207cf bytes) 000184 1772435263.163075 MIRROR_EXTEND to 2 mirrors 000185 1772435264.187815 write 0x0044b5 thru 0x00fe7f (0xb9cb bytes) 000186 1772435264.258789 trunc from 0x01f831 to 0x037d49 (0x18519 bytes) 000187 1772435264.283698 mapread 0x02c992 thru 0x036e66 (0xa4d5 bytes) 000188 1772435264.313063 mapread 0x001b27 thru 0x005d8e (0x4268 bytes) 000189 1772435264.324132 mapwrite 0x00e526 thru 0x01cbfd (0xe6d8 bytes) 000190 1772435264.396838 trunc from 0x037d4a to 0x02db14 (0xa235 bytes) 000194 1772435264.416036 mapread 0x01766c thru 0x0196c4 (0x2059 bytes) 000195 1772435264.418363 mapwrite 0x02e101 thru 0x03820e (0xa10e bytes) 000200 1772435264.516464 read 0x00793d thru 0x00e64c (0x6d10 bytes) 000201 1772435264.519382 mapwrite 0x0113d1 thru 0x01c31d (0xaf4d bytes) 000203 1772435264.605524 MIRROR_RESYNC 2 mirrors 000204 1772435265.420006 mapwrite 0x0049de thru 0x012f65 (0xe588 bytes) 000205 1772435265.604790 mapwrite 0x00fe3a thru 0x01a499 (0xa660 bytes) 000206 1772435265.679905 mapread 0x012802 thru 0x01efcf (0xc7ce bytes) 000207 1772435265.682802 mapread 0x009669 thru 0x00c4a8 (0x2e40 bytes) 000208 1772435265.689475 MIRROR_RESYNC 2 mirrors 000209 1772435266.315548 read 0x030d83 thru 0x035888 (0x4b06 bytes) 000210 1772435266.322496 mapread 0x01642d thru 0x02084a (0xa41e bytes) 000211 1772435266.364953 mapread 0x010f03 thru 0x013c93 (0x2d91 bytes) 000212 1772435266.374000 trunc from 0x03820f to 0x03838b (0x17d bytes) 000219 1772435266.493472 write 0x039ca7 thru 0x03ffff (0x6359 bytes) 000222 1772435266.497125 trunc from 0x040000 to 0x030c98 (0xf367 bytes) 000224 1772435266.527005 MIRROR_RESYNC 2 mirrors 000225 1772435267.097078 read 0x0250e7 thru 0x026282 (0x119c bytes) 000226 1772435267.103850 trunc from 0x030c99 to 0x03b047 (0xa3af bytes) 000227 1772435267.206863 trunc from 0x03b048 to 0x007bf6 (0x33451 bytes) 000228 1772435267.274383 trunc from 0x007bf7 to 0x02d868 (0x25c72 bytes) 000230 1772435267.326115 mapwrite 0x0069f1 thru 0x00ff16 (0x9526 bytes) 000232 1772435267.497093 write 0x01cb75 thru 0x01e324 (0x17b0 bytes) 000236 1772435267.516857 mapwrite 0x026f0a thru 0x0306cb (0x97c2 bytes) 000237 1772435267.638269 mapwrite 0x00129c thru 0x010848 (0xf5ad bytes) 000238 1772435267.798832 write 0x015732 thru 0x021a2a (0xc2f9 bytes) 000240 1772435267.813138 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x306cc): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x306cc): Mirror 2: 0x74463ebc (stale) CRC-32 checksum value for chunk [0, 0x306cc): Mirror 1: 0x75fecf19 Mirror 3: 0x75fecf19 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x306cc: skipped 000242 1772435269.070140 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) 000243 1772435269.571753 read 0x02a63c thru 0x02c3e6 (0x1dab bytes) 000246 1772435269.643467 read 0x02bbff thru 0x03028b (0x468d bytes) 000247 1772435269.682331 mapwrite 0x00ab80 thru 0x013a58 (0x8ed9 bytes) 000248 1772435269.967479 mapwrite 0x030503 thru 0x03d5d1 (0xd0cf bytes) 000251 1772435270.136961 mapread 0x017fbc thru 0x01e67a (0x66bf bytes) 000252 1772435270.205227 MIRROR_RESYNC 2 mirrors 000253 1772435271.073734 mapread 0x01fbef thru 0x02e0bd (0xe4cf bytes) 000254 1772435271.119476 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) 000255 1772435271.310621 read 0x0226ae thru 0x027d8a (0x56dd bytes) 000257 1772435271.337999 read 0x00729c thru 0x00dd05 (0x6a6a bytes) 000258 1772435271.384259 mapread 0x00ff54 thru 0x0103c4 (0x471 bytes) 000264 1772435271.397518 mapwrite 0x0025af thru 0x008cfe (0x6750 bytes) 000273 1772435271.551272 write 0x00befe thru 0x011972 (0x5a75 bytes) 000275 1772435271.569301 MIRROR_EXTEND to 2 mirrors 000276 1772435272.325454 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) 000278 1772435272.495442 mapread 0x01846a thru 0x01a4c7 (0x205e bytes) 000279 1772435272.516431 read 0x00731d thru 0x015d14 (0xe9f8 bytes) 000280 1772435272.527475 mapwrite 0x03ce52 thru 0x03ef22 (0x20d1 bytes) 000281 1772435272.642893 mapread 0x003f0f thru 0x012143 (0xe235 bytes) 000284 1772435272.663908 read 0x02dae8 thru 0x0354fc (0x7a15 bytes) 000286 1772435272.676320 write 0x005fc5 thru 0x00c51c (0x6558 bytes) 000292 1772435272.682009 read 0x03c761 thru 0x03ef22 (0x27c2 bytes) 000295 1772435272.684528 trunc from 0x03ef23 to 0x00732e (0x37bf4 bytes) 000296 1772435272.707268 read 0x005eda thru 0x00732e (0x1455 bytes) 000297 1772435272.708132 read 0x0058e5 thru 0x00732e (0x1a4a bytes) 000298 1772435272.710086 trunc from 0x00732f to 0x030b02 (0x297d4 bytes) 000303 1772435272.735056 read 0x029cec thru 0x02e08b (0x43a0 bytes) 000305 1772435272.757760 write 0x02cade thru 0x03bc25 (0xf148 bytes) 000309 1772435272.761942 read 0x01a854 thru 0x01e117 (0x38c4 bytes) 000311 1772435272.763009 write 0x02c2fe thru 0x032d70 (0x6a73 bytes) 000317 1772435272.766327 mapread 0x0118ae thru 0x01323d (0x1990 bytes) 000318 1772435272.767819 mapwrite 0x0353e2 thru 0x03ffff (0xac1e bytes) 000320 1772435272.855322 trunc from 0x040000 to 0x0312ce (0xed31 bytes) 000321 1772435272.875447 mapread 0x014941 thru 0x02385c (0xef1c bytes) 000322 1772435272.876915 mapread 0x028325 thru 0x0312ce (0x8faa bytes) 000323 1772435272.879479 mapwrite 0x020f55 thru 0x02ebf0 (0xdc9c bytes) 000324 1772435272.932931 write 0x005b25 thru 0x01263b (0xcb17 bytes) 000325 1772435272.934302 trunc from 0x0312cf to 0x02e6a7 (0x2c27 bytes) 000326 1772435272.954179 write 0x009058 thru 0x009b50 (0xaf9 bytes) 000329 1772435272.962557 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x3c98e 000331 1772435273.802610 trunc from 0x02e6a8 to 0x03c98d (0xe2e6 bytes) 000332 1772435273.965500 read 0x014d12 thru 0x01b2ad (0x659c bytes) 000333 1772435273.976974 write 0x0184e5 thru 0x020fda (0x8af6 bytes) 000334 1772435273.985968 read 0x027659 thru 0x02cd29 (0x56d1 bytes) 000341 1772435274.009536 mapwrite 0x0124d9 thru 0x0152de (0x2e06 bytes) 000342 1772435274.060983 write 0x0179be thru 0x01f9a7 (0x7fea bytes) 000344 1772435274.063509 mapread 0x02f160 thru 0x039ea0 (0xad41 bytes) 000345 1772435274.066000 read 0x015199 thru 0x0218a4 (0xc70c bytes) 000347 1772435274.068687 trunc from 0x03c98e to 0x034021 (0x896c bytes) 000350 1772435274.094102 mapwrite 0x01eed8 thru 0x02b3ed (0xc516 bytes) 000353 1772435274.147379 mapwrite 0x02549a thru 0x034c2e (0xf795 bytes) 000354 1772435274.227190 read 0x0003e4 thru 0x00b9e8 (0xb605 bytes) 000360 1772435274.239201 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x34c2f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34c2f): Mirror 2: 0xe64e716 (stale) CRC-32 checksum value for chunk [0, 0x34c2f): Mirror 1: 0xcd0c4687 Mirror 3: 0xcd0c4687 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34c2f: skipped 000361 1772435275.168034 mapread 0x001c2e thru 0x00356f (0x1942 bytes) 000362 1772435275.192782 mapwrite 0x0294b0 thru 0x0359fb (0xc54c bytes) 000364 1772435275.418962 trunc from 0x0359fc to 0x000c62 (0x34d99 bytes) 000367 1772435275.475522 trunc from 0x000c63 to 0x0367c8 (0x35b66 bytes) 000370 1772435275.520510 mapread 0x0250f9 thru 0x0337bd (0xe6c5 bytes) 000371 1772435275.557645 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x367c9): Mirror 3: 0x30be057a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x367c9): Mirror 3: 0x30be057a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x367c9: skipped 000372 1772435275.874238 trunc from 0x0367c9 to 0x0114bb (0x2530d bytes) 000374 1772435275.906989 trunc from 0x0114bc to 0x02c941 (0x1b486 bytes) 000376 1772435275.931748 MIRROR_RESYNC 2 mirrors 000378 1772435276.829890 mapread 0x00adaf thru 0x018bfa (0xde4c bytes) 000379 1772435276.879928 trunc from 0x02c942 to 0x011a83 (0x1aebe bytes) 000382 1772435277.019642 mapwrite 0x01030e thru 0x01f541 (0xf234 bytes) 000385 1772435277.177479 mapwrite 0x017b75 thru 0x0269f4 (0xee80 bytes) 000388 1772435277.362959 read 0x003968 thru 0x00892d (0x4fc6 bytes) 000389 1772435277.388493 MIRROR_RESYNC 2 mirrors 000391 1772435278.029510 write 0x01e32d thru 0x0217f7 (0x34cb bytes) 000392 1772435278.163169 mapread 0x00241c thru 0x00bdf5 (0x99da bytes) 000393 1772435278.182919 write 0x03eaa8 thru 0x03ffff (0x1558 bytes) 000394 1772435278.185454 mapread 0x000345 thru 0x007256 (0x6f12 bytes) 000395 1772435278.198482 mapwrite 0x03a42d thru 0x03ffff (0x5bd3 bytes) 000396 1772435278.308904 trunc from 0x040000 to 0x01c77c (0x23883 bytes) 000397 1772435278.420314 mapwrite 0x03f539 thru 0x03ffff (0xac7 bytes) 000399 1772435278.539510 read 0x005a6b thru 0x013f2c (0xe4c2 bytes) 000400 1772435278.545892 read 0x00ced2 thru 0x011f9f (0x50ce bytes) 000401 1772435278.550852 MIRROR_RESYNC 2 mirrors 000402 1772435279.217725 MIRROR_EXTEND to 3 mirrors 000403 1772435279.795939 write 0x038892 thru 0x03d9cf (0x513e bytes) 000404 1772435279.871274 read 0x0241df thru 0x024556 (0x378 bytes) 000406 1772435279.878806 read 0x0203c2 thru 0x02793a (0x7579 bytes) 000407 1772435279.892866 MIRROR_RESYNC 3 mirrors 000408 1772435280.727884 trunc from 0x040000 to 0x015360 (0x2ac9f bytes) 000410 1772435280.780692 trunc from 0x015361 to 0x00588d (0xfad3 bytes) 000411 1772435280.815926 trunc from 0x00588e to 0x01d02e (0x177a1 bytes) 000412 1772435280.831135 mapwrite 0x01afa4 thru 0x0295bf (0xe61c bytes) 000415 1772435280.936346 trunc from 0x0295c0 to 0x026e64 (0x275b bytes) 000417 1772435280.959042 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x26e65): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26e65): Mirror 3: 0x60411f02 (stale) Mirror 4: 0x60411f02 (stale) CRC-32 checksum value for chunk [0, 0x26e65): Mirror 1: 0x1b4320c3 Mirror 5: 0x1b4320c3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26e65: skipped 000418 1772435281.725866 mapread 0x01fbc8 thru 0x026098 (0x64d1 bytes) 000419 1772435281.757277 mapwrite 0x03fd97 thru 0x03ffff (0x269 bytes) 000422 1772435281.843009 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7807543d (stale) Mirror 4: 0x7807543d (stale) Mirror 5: 0x8adf7d93 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xef1a8c1e Mirror 6: 0xef1a8c1e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000424 1772435282.596826 mapwrite 0x03eddf thru 0x03ffff (0x1221 bytes) 000425 1772435282.835506 mapread 0x012992 thru 0x01a019 (0x7688 bytes) 000426 1772435282.853784 mapwrite 0x029d4d thru 0x035dd5 (0xc089 bytes) 000431 1772435282.973289 read 0x03ae7c thru 0x03e8c5 (0x3a4a bytes) 000432 1772435282.981062 mapwrite 0x00e927 thru 0x0198a4 (0xaf7e bytes) 000434 1772435283.089311 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x7807543d (stale) Mirror 5: 0x8adf7d93 (stale) Mirror 6: 0xef1a8c1e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x7807543d (stale) Mirror 5: 0x8adf7d93 (stale) Mirror 6: 0xef1a8c1e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000436 1772435283.565420 write 0x0260e8 thru 0x02d6e8 (0x7601 bytes) 000437 1772435283.583857 trunc from 0x040000 to 0x00a612 (0x359ed bytes) 000439 1772435283.617407 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xa613): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa613): Mirror 4: 0xc5c36c7b (stale) Mirror 5: 0xc5c36c7b (stale) Mirror 6: 0xc5c36c7b (stale) CRC-32 checksum value for chunk [0, 0xa613): Mirror 1: 0xc5c36c7b Mirror 7: 0xc5c36c7b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa613: skipped 000443 1772435284.705339 mapwrite 0x03b5c6 thru 0x03ffff (0x4a3a bytes) 000444 1772435285.037008 mapread 0x00b724 thru 0x017734 (0xc011 bytes) 000447 1772435285.101487 mapwrite 0x011dbe thru 0x013d21 (0x1f64 bytes) 000448 1772435285.250617 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x7807543d (stale) Mirror 5: 0x8adf7d93 (stale) Mirror 6: 0xef1a8c1e (stale) Mirror 7: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8de473a6 Mirror 8: 0x8de473a6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000450 1772435286.690988 mapwrite 0x01e5de thru 0x023fb2 (0x59d5 bytes) 000451 1772435286.959514 mapread 0x024c89 thru 0x032a8a (0xde02 bytes) 000452 1772435286.990286 mapwrite 0x03c913 thru 0x03ff65 (0x3653 bytes) 000454 1772435287.078009 mapread 0x026ed5 thru 0x02f4fb (0x8627 bytes) 000457 1772435287.084272 write 0x033679 thru 0x03a2ae (0x6c36 bytes) 000458 1772435287.087135 mapwrite 0x03a6ad thru 0x03f354 (0x4ca8 bytes) 000460 1772435287.141308 read 0x001957 thru 0x00e769 (0xce13 bytes) 000461 1772435287.156557 MIRROR_RESYNC 6 mirrors 000463 1772435288.963709 write 0x024174 thru 0x0321c0 (0xe04d bytes) 000466 1772435289.069457 trunc from 0x040000 to 0x0127c0 (0x2d83f bytes) 000468 1772435289.119624 mapwrite 0x01fe10 thru 0x02b8cd (0xbabe bytes) 000469 1772435289.394004 trunc from 0x02b8ce to 0x002abb (0x28e12 bytes) 000470 1772435289.425788 write 0x0382e7 thru 0x03ffff (0x7d19 bytes) 000474 1772435289.436287 write 0x0099e3 thru 0x00afef (0x160d bytes) 000475 1772435289.463992 read 0x0265ef thru 0x02d7de (0x71f0 bytes) 000476 1772435289.482388 mapread 0x03f927 thru 0x03ffff (0x6d9 bytes) 000479 1772435289.492293 mapread 0x0330e0 thru 0x03cafe (0x9a1f bytes) 000481 1772435289.527347 MIRROR_RESYNC 6 mirrors 000482 1772435290.615377 mapwrite 0x01b6f5 thru 0x020d90 (0x569c bytes) 000485 1772435290.758294 mapread 0x024682 thru 0x02777d (0x30fc bytes) 000488 1772435290.770826 mapread 0x019781 thru 0x01d416 (0x3c96 bytes) 000489 1772435290.772202 write 0x017718 thru 0x026116 (0xe9ff bytes) 000491 1772435290.775542 trunc from 0x040000 to 0x0013f4 (0x3ec0b bytes) 000493 1772435290.801987 mapread 0x000fc5 thru 0x0013f4 (0x430 bytes) 000495 1772435290.818944 mapread 0x00092b thru 0x0013f4 (0xaca bytes) 000496 1772435290.827336 trunc from 0x0013f5 to 0x0181dd (0x16de9 bytes) 000502 1772435290.873076 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x181de): Mirror 5: 0x5c83c596 (stale) Mirror 6: 0x5c83c596 (stale) Mirror 7: 0x5c83c596 (stale) Mirror 8: 0x5c83c596 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x181de): Mirror 5: 0x5c83c596 (stale) Mirror 6: 0x5c83c596 (stale) Mirror 7: 0x5c83c596 (stale) Mirror 8: 0x5c83c596 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x181de: skipped 000504 1772435291.216331 trunc from 0x0181de to 0x01858b (0x3ae bytes) 000506 1772435291.228088 write 0x008fb8 thru 0x00f7ab (0x67f4 bytes) 000507 1772435291.237544 mapread 0x013920 thru 0x01858b (0x4c6c bytes) 000510 1772435291.249281 write 0x004981 thru 0x008e10 (0x4490 bytes) 000511 1772435291.260010 mapwrite 0x036d76 thru 0x03ffff (0x928a bytes) 000512 1772435291.422519 mapread 0x0097c3 thru 0x016621 (0xce5f bytes) 000513 1772435291.425757 mapread 0x035f11 thru 0x03e8cd (0x89bd bytes) 000515 1772435291.430402 mapwrite 0x010aa5 thru 0x01b40e (0xa96a bytes) 000516 1772435291.513492 trunc from 0x040000 to 0x00a92a (0x356d5 bytes) 000517 1772435291.547359 MIRROR_RESYNC 5 mirrors 000518 1772435293.174081 mapwrite 0x02fef6 thru 0x038f07 (0x9012 bytes) 000523 1772435293.341741 read 0x01bada thru 0x024459 (0x8980 bytes) 000525 1772435293.351652 mapwrite 0x00f52b thru 0x01ef0b (0xf9e1 bytes) 000527 1772435293.416851 mapread 0x009bdc thru 0x01274e (0x8b73 bytes) 000529 1772435293.420104 read 0x01d415 thru 0x02b965 (0xe551 bytes) 000530 1772435293.430792 write 0x02ffe4 thru 0x0300c1 (0xde bytes) 000532 1772435293.439028 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38f08): Mirror 6: 0xab1fd806 (stale) Mirror 7: 0xab1fd806 (stale) Mirror 8: 0xab1fd806 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f08): Mirror 6: 0xab1fd806 (stale) Mirror 7: 0xab1fd806 (stale) Mirror 8: 0xab1fd806 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f08: skipped 000533 1772435294.039682 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x38f08): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f08): Mirror 6: 0xab1fd806 (stale) Mirror 7: 0xab1fd806 (stale) Mirror 8: 0xab1fd806 (stale) CRC-32 checksum value for chunk [0, 0x38f08): Mirror 1: 0xa68428a3 Mirror 9: 0xa68428a3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f08: skipped 000534 1772435295.387323 mapwrite 0x02ea70 thru 0x03d089 (0xe61a bytes) 000535 1772435295.638347 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d08a): Mirror 6: 0x893c11ff (stale) Mirror 7: 0x893c11ff (stale) Mirror 9: 0x6ea10248 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d08a): Mirror 6: 0x893c11ff (stale) Mirror 7: 0x893c11ff (stale) Mirror 9: 0x6ea10248 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d08a: skipped 000538 1772435296.165262 read 0x024094 thru 0x032df0 (0xed5d bytes) 000539 1772435296.203692 mapwrite 0x023a22 thru 0x02f83b (0xbe1a bytes) 000540 1772435296.376649 mapwrite 0x01014d thru 0x011b50 (0x1a04 bytes) 000542 1772435296.502319 trunc from 0x03d08a to 0x0251c9 (0x17ec0 bytes) 000543 1772435296.538626 MIRROR_RESYNC 4 mirrors 000544 1772435297.557086 mapwrite 0x02916e thru 0x037549 (0xe3dc bytes) 000546 1772435297.854842 trunc from 0x03754a to 0x003891 (0x33cb8 bytes) 000547 1772435297.905044 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3892): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3892): Mirror 6: 0xa7588f1 (stale) Mirror 7: 0xa7588f1 (stale) Mirror 9: 0xa7588f1 (stale) CRC-32 checksum value for chunk [0, 0x3892): Mirror 1: 0xa7588f1 Mirror 10: 0xa7588f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3892: skipped 000550 1772435299.239855 read 0x00252d thru 0x003891 (0x1365 bytes) 000551 1772435299.259454 trunc from 0x003892 to 0x02d5b3 (0x29d22 bytes) 000552 1772435299.412143 write 0x00f335 thru 0x011700 (0x23cc bytes) 000554 1772435299.424483 mapread 0x021231 thru 0x02134a (0x11a bytes) 000556 1772435299.439880 write 0x02ca16 thru 0x03c130 (0xf71b bytes) 000557 1772435299.449534 write 0x03a7cf thru 0x03b427 (0xc59 bytes) 000558 1772435299.451679 mapread 0x03678b thru 0x03c130 (0x59a6 bytes) 000562 1772435299.455688 mapread 0x01c5bc thru 0x020a0a (0x444f bytes) 000563 1772435299.457852 read 0x006bf0 thru 0x015938 (0xed49 bytes) 000564 1772435299.470838 mapread 0x01485e thru 0x0174c0 (0x2c63 bytes) 000567 1772435299.474754 trunc from 0x03c131 to 0x03c7fe (0x6ce bytes) 000569 1772435299.515754 MIRROR_RESYNC 5 mirrors 000571 1772435301.061851 mapwrite 0x02a21a thru 0x02d871 (0x3658 bytes) 000572 1772435301.323299 trunc from 0x03c7ff to 0x0075e7 (0x35217 bytes) 000573 1772435301.392736 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x75e8): Mirror 6: 0xfdbb0d20 (stale) Mirror 7: 0xfdbb0d20 (stale) Mirror 10: 0xfdbb0d20 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x75e8): Mirror 6: 0xfdbb0d20 (stale) Mirror 7: 0xfdbb0d20 (stale) Mirror 10: 0xfdbb0d20 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x75e8: skipped 000574 1772435301.947364 read 0x002b59 thru 0x0075e7 (0x4a8f bytes) 000575 1772435301.956696 write 0x03b403 thru 0x03e895 (0x3493 bytes) 000576 1772435301.958081 mapread 0x01c9ad thru 0x021e3e (0x5492 bytes) 000577 1772435301.981197 MIRROR_RESYNC 4 mirrors 000579 1772435303.180489 read 0x016308 thru 0x020131 (0x9e2a bytes) 000580 1772435303.189141 write 0x0304fe thru 0x035e74 (0x5977 bytes) 000581 1772435303.275962 mapwrite 0x024652 thru 0x02a074 (0x5a23 bytes) 000582 1772435303.374864 read 0x03e764 thru 0x03e895 (0x132 bytes) 000584 1772435303.429386 MIRROR_RESYNC 4 mirrors 000587 1772435304.466139 trunc from 0x03e896 to 0x036d08 (0x7b8d bytes) 000588 1772435304.551258 mapwrite 0x02e8bb thru 0x035e32 (0x7578 bytes) 000589 1772435304.635137 write 0x03a23b thru 0x03ffff (0x5dc5 bytes) 000590 1772435304.637632 read 0x029162 thru 0x0371f9 (0xe098 bytes) 000591 1772435304.645068 trunc from 0x040000 to 0x01f15b (0x20ea4 bytes) 000593 1772435304.678688 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1f15c): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f15c): Mirror 6: 0x5794368c (stale) Mirror 7: 0x5794368c (stale) Mirror 10: 0x5794368c (stale) CRC-32 checksum value for chunk [0, 0x1f15c): Mirror 1: 0x5794368c Mirror 11: 0x5794368c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f15c: skipped 000594 1772435305.515166 read 0x0198b3 thru 0x01f15b (0x58a9 bytes) 000595 1772435305.532735 write 0x015160 thru 0x018a17 (0x38b8 bytes) 000596 1772435305.628180 read 0x0053e7 thru 0x011cb8 (0xc8d2 bytes) 000597 1772435305.640019 trunc from 0x01f15c to 0x00f658 (0xfb03 bytes) 000598 1772435305.659550 mapread 0x0032a9 thru 0x00494f (0x16a7 bytes) 000603 1772435305.665335 trunc from 0x00f659 to 0x029b00 (0x1a4a8 bytes) 000604 1772435305.687720 MIRROR_RESYNC 5 mirrors 000605 1772435306.525384 trunc from 0x029b01 to 0x01e2ba (0xb846 bytes) 000608 1772435306.665798 mapwrite 0x02f69a thru 0x032828 (0x318f bytes) 000609 1772435306.800964 trunc from 0x032829 to 0x0358ac (0x3084 bytes) 000610 1772435306.825074 mapwrite 0x023791 thru 0x024a1d (0x128d bytes) 000611 1772435306.890188 read 0x021dd7 thru 0x02809d (0x62c7 bytes) 000613 1772435306.892877 mapwrite 0x0074f1 thru 0x012565 (0xb075 bytes) 000616 1772435306.991437 mapread 0x0123a7 thru 0x021840 (0xf49a bytes) 000619 1772435307.000340 mapwrite 0x03c393 thru 0x03ffff (0x3c6d bytes) 000620 1772435307.114698 read 0x0152a1 thru 0x019c61 (0x49c1 bytes) 000621 1772435307.123101 MIRROR_RESYNC 5 mirrors 000622 1772435308.084225 read 0x018fb8 thru 0x020505 (0x754e bytes) 000623 1772435308.094442 read 0x015e85 thru 0x01f449 (0x95c5 bytes) 000624 1772435308.106462 trunc from 0x040000 to 0x02d477 (0x12b88 bytes) 000627 1772435308.220200 trunc from 0x02d478 to 0x01b47c (0x11ffb bytes) 000628 1772435308.240971 mapwrite 0x02474f thru 0x02e527 (0x9dd9 bytes) 000630 1772435308.384174 write 0x035619 thru 0x037d87 (0x276f bytes) 000631 1772435308.391481 MIRROR_RESYNC 5 mirrors 000633 1772435309.436536 mapwrite 0x03e6f7 thru 0x03ffff (0x1909 bytes) 000635 1772435309.551441 mapread 0x016656 thru 0x019248 (0x2bf3 bytes) 000636 1772435309.562067 mapwrite 0x03b766 thru 0x03ffff (0x489a bytes) 000639 1772435309.621419 write 0x009644 thru 0x009da8 (0x765 bytes) 000640 1772435309.635536 mapwrite 0x004b45 thru 0x01009c (0xb558 bytes) 000642 1772435309.699220 mapwrite 0x027a0f thru 0x0365d2 (0xebc4 bytes) 000643 1772435309.786655 mapread 0x0170ad thru 0x0259d1 (0xe925 bytes) 000644 1772435309.793262 mapwrite 0x00ece6 thru 0x019b72 (0xae8d bytes) 000646 1772435309.856206 trunc from 0x040000 to 0x00201d (0x3dfe2 bytes) 000649 1772435309.890298 write 0x027fda thru 0x02e531 (0x6558 bytes) 000652 1772435309.895162 mapwrite 0x029f7f thru 0x032fd0 (0x9052 bytes) 000653 1772435309.993516 read 0x01078d thru 0x019f84 (0x97f8 bytes) 000655 1772435310.016408 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32fd1): Mirror 6: 0x7f7c90bd (stale) Mirror 10: 0x7f7c90bd (stale) Mirror 11: 0x7f7c90bd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32fd1): Mirror 6: 0x7f7c90bd (stale) Mirror 10: 0x7f7c90bd (stale) Mirror 11: 0x7f7c90bd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32fd1: skipped 000656 1772435310.368296 write 0x022a2b thru 0x032660 (0xfc36 bytes) 000658 1772435310.387488 read 0x012b4e thru 0x01fcef (0xd1a2 bytes) 000660 1772435310.396805 mapwrite 0x02e686 thru 0x039078 (0xa9f3 bytes) 000661 1772435310.473521 read 0x0233b3 thru 0x02a416 (0x7064 bytes) 000663 1772435310.474983 mapread 0x02e71f thru 0x030ee6 (0x27c8 bytes) 000665 1772435310.477936 mapread 0x03303e thru 0x036279 (0x323c bytes) 000666 1772435310.479768 read 0x01c70b thru 0x02265c (0x5f52 bytes) 000667 1772435310.486737 mapwrite 0x010061 thru 0x0146a3 (0x4643 bytes) 000671 1772435310.531044 write 0x039be7 thru 0x03ffff (0x6419 bytes) 000675 1772435310.537497 write 0x018b5f thru 0x01983a (0xcdc bytes) 000676 1772435310.539451 write 0x01e986 thru 0x02a20b (0xb886 bytes) 000677 1772435310.541768 trunc from 0x040000 to 0x03b941 (0x46be bytes) 000678 1772435310.561047 write 0x02a228 thru 0x039fb6 (0xfd8f bytes) 000680 1772435310.564040 trunc from 0x03b942 to 0x020866 (0x1b0db bytes) 000681 1772435310.585240 read 0x01dc3f thru 0x020866 (0x2c28 bytes) 000682 1772435310.594190 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x20867): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20867): Mirror 6: 0xb5e948da (stale) Mirror 10: 0xb5e948da (stale) Mirror 11: 0xb5e948da (stale) CRC-32 checksum value for chunk [0, 0x20867): Mirror 1: 0xe3d05c19 Mirror 12: 0xe3d05c19 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20867: skipped 000683 1772435311.828733 mapread 0x01200d thru 0x0204bd (0xe4b1 bytes) 000685 1772435311.876036 mapread 0x0103d3 thru 0x010c5d (0x88b bytes) 000687 1772435311.896507 write 0x038bbe thru 0x03ab3e (0x1f81 bytes) 000690 1772435311.999763 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3ab3f): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ab3f): Mirror 6: 0xc48b7de3 (stale) Mirror 10: 0xc48b7de3 (stale) Mirror 11: 0xc48b7de3 (stale) Mirror 12: 0xd7c64e6a (stale) CRC-32 checksum value for chunk [0, 0x3ab3f): Mirror 1: 0xcbb1a789 Mirror 13: 0xcbb1a789 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ab3f: skipped 000691 1772435313.200143 read 0x00e7b2 thru 0x019c6f (0xb4be bytes) 000693 1772435313.274243 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3ab3f): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ab3f): Mirror 6: 0xc48b7de3 (stale) Mirror 10: 0xc48b7de3 (stale) Mirror 11: 0xc48b7de3 (stale) Mirror 12: 0xd7c64e6a (stale) CRC-32 checksum value for chunk [0, 0x3ab3f): Mirror 1: 0xcbb1a789 Mirror 13: 0xcbb1a789 Mirror 14: 0xcbb1a789 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ab3f: skipped 000694 1772435314.656863 read 0x039b46 thru 0x03ab3e (0xff9 bytes) 000697 1772435314.682620 mapread 0x0361f0 thru 0x03ab3e (0x494f bytes) 000700 1772435314.715618 trunc from 0x03ab3f to 0x02485a (0x162e4 bytes) 000702 1772435314.823165 write 0x02e67e thru 0x038b78 (0xa4fb bytes) 000703 1772435314.827919 mapread 0x012209 thru 0x01e6de (0xc4d6 bytes) 000704 1772435314.861555 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x38b79): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38b79): Mirror 6: 0x591036b5 (stale) Mirror 10: 0x591036b5 (stale) Mirror 11: 0x591036b5 (stale) Mirror 12: 0xc75229ce (stale) Mirror 13: 0xc75229ce (stale) Mirror 14: 0xc75229ce (stale) CRC-32 checksum value for chunk [0, 0x38b79): Mirror 1: 0xc9ca4f12 Mirror 15: 0xc9ca4f12 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38b79: skipped 000705 1772435316.033668 mapread 0x0315e7 thru 0x036b6f (0x5589 bytes) 000706 1772435316.080111 write 0x01702b thru 0x024987 (0xd95d bytes) 000707 1772435316.193531 read 0x002174 thru 0x00ce0c (0xac99 bytes) 000709 1772435316.212490 write 0x004dcd thru 0x007988 (0x2bbc bytes) 000713 1772435316.218799 read 0x01107c thru 0x011181 (0x106 bytes) 000714 1772435316.223093 mapread 0x025cfe thru 0x02b3e7 (0x56ea bytes) 000718 1772435316.253781 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x38b79): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38b79): Mirror 6: 0x591036b5 (stale) Mirror 10: 0x591036b5 (stale) Mirror 11: 0x591036b5 (stale) Mirror 12: 0xc75229ce (stale) Mirror 13: 0xc75229ce (stale) Mirror 14: 0xc75229ce (stale) Mirror 15: 0xc9ca4f12 (stale) CRC-32 checksum value for chunk [0, 0x38b79): Mirror 1: 0x77e2415f Mirror 16: 0x77e2415f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38b79: skipped 000719 1772435317.610122 read 0x02e95a thru 0x033a0b (0x50b2 bytes) 000720 1772435317.627468 read 0x0209b7 thru 0x02d8e7 (0xcf31 bytes) 000721 1772435317.650605 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38b79): Mirror 6: 0x591036b5 (stale) Mirror 10: 0x591036b5 (stale) Mirror 12: 0xc75229ce (stale) Mirror 13: 0xc75229ce (stale) Mirror 14: 0xc75229ce (stale) Mirror 15: 0xc9ca4f12 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38b79): Mirror 6: 0x591036b5 (stale) Mirror 10: 0x591036b5 (stale) Mirror 12: 0xc75229ce (stale) Mirror 13: 0xc75229ce (stale) Mirror 14: 0xc75229ce (stale) Mirror 15: 0xc9ca4f12 (stale) CRC-32 checksum value for chunk [0, 0x38b79): Mirror 1: 0x77e2415f Mirror 16: 0x77e2415f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38b79: skipped 000723 1772435318.743468 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x38b79): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38b79): Mirror 6: 0x591036b5 (stale) Mirror 10: 0x591036b5 (stale) Mirror 12: 0xc75229ce (stale) Mirror 13: 0xc75229ce (stale) Mirror 14: 0xc75229ce (stale) Mirror 15: 0xc9ca4f12 (stale) CRC-32 checksum value for chunk [0, 0x38b79): Mirror 1: 0x77e2415f Mirror 16: 0x77e2415f Mirror 17: 0x77e2415f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38b79: skipped 000724 1772435320.386500 mapwrite 0x01df29 thru 0x0234f0 (0x55c8 bytes) 000725 1772435320.625936 write 0x018e49 thru 0x024ddc (0xbf94 bytes) 000726 1772435320.627388 mapread 0x009a4e thru 0x00abbd (0x1170 bytes) 000727 1772435320.638971 mapwrite 0x03c0a8 thru 0x03ffff (0x3f58 bytes) 000729 1772435320.752760 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe01273d0 (stale) Mirror 10: 0xe01273d0 (stale) Mirror 12: 0xd992ddac (stale) Mirror 13: 0xcded9169 (stale) Mirror 14: 0xcded9169 (stale) Mirror 15: 0x6004ec2c (stale) Mirror 17: 0xa16b8d39 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe01273d0 (stale) Mirror 10: 0xe01273d0 (stale) Mirror 12: 0xd992ddac (stale) Mirror 13: 0xcded9169 (stale) Mirror 14: 0xcded9169 (stale) Mirror 15: 0x6004ec2c (stale) Mirror 17: 0xa16b8d39 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000730 1772435321.238348 mapwrite 0x02b869 thru 0x031d1f (0x64b7 bytes) 000731 1772435321.342682 mapwrite 0x017bf9 thru 0x02383c (0xbc44 bytes) 000732 1772435321.482378 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe01273d0 (stale) Mirror 10: 0xe01273d0 (stale) Mirror 12: 0xd992ddac (stale) Mirror 13: 0xcded9169 (stale) Mirror 14: 0xcded9169 (stale) Mirror 15: 0x6004ec2c (stale) Mirror 17: 0xa16b8d39 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xad56af8c Mirror 18: 0xad56af8c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000733 1772435322.934083 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe01273d0 (stale) Mirror 10: 0xe01273d0 (stale) Mirror 12: 0xd992ddac (stale) Mirror 13: 0xcded9169 (stale) Mirror 14: 0xcded9169 (stale) Mirror 15: 0x6004ec2c (stale) Mirror 17: 0xa16b8d39 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe01273d0 (stale) Mirror 10: 0xe01273d0 (stale) Mirror 12: 0xd992ddac (stale) Mirror 13: 0xcded9169 (stale) Mirror 14: 0xcded9169 (stale) Mirror 15: 0x6004ec2c (stale) Mirror 17: 0xa16b8d39 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000734 1772435323.627197 write 0x037341 thru 0x03ffff (0x8cbf bytes) 000735 1772435323.645899 mapread 0x0006be thru 0x00d976 (0xd2b9 bytes) 000736 1772435323.668278 mapwrite 0x0151eb thru 0x01f6a3 (0xa4b9 bytes) 000739 1772435323.799982 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe01273d0 (stale) Mirror 10: 0xe01273d0 (stale) Mirror 13: 0xcded9169 (stale) Mirror 14: 0xcded9169 (stale) Mirror 15: 0x6004ec2c (stale) Mirror 17: 0xa16b8d39 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe01273d0 (stale) Mirror 10: 0xe01273d0 (stale) Mirror 13: 0xcded9169 (stale) Mirror 14: 0xcded9169 (stale) Mirror 15: 0x6004ec2c (stale) Mirror 17: 0xa16b8d39 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000741 1772435324.372367 trunc from 0x040000 to 0x036a42 (0x95bd bytes) 000742 1772435324.396522 write 0x024dd1 thru 0x0330cf (0xe2ff bytes) 000748 1772435324.415554 read 0x0148be thru 0x01db15 (0x9258 bytes) 000749 1772435324.429056 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x36a43): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36a43): Mirror 6: 0x936fc56 (stale) Mirror 10: 0x936fc56 (stale) Mirror 13: 0xed86cf04 (stale) Mirror 14: 0xed86cf04 (stale) Mirror 15: 0x8953e89 (stale) Mirror 17: 0x3535bd24 (stale) CRC-32 checksum value for chunk [0, 0x36a43): Mirror 1: 0xfab7df78 Mirror 18: 0xfab7df78 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36a43: skipped 000751 1772435325.576023 read 0x02e13b thru 0x036a42 (0x8908 bytes) 000753 1772435325.601531 MIRROR_RESYNC 8 mirrors 000755 1772435327.114338 MIRROR_EXTEND to 9 mirrors 000756 1772435327.987128 trunc from 0x036a43 to 0x004d5f (0x31ce3 bytes) 000757 1772435328.118180 mapwrite 0x00b0c4 thru 0x00b761 (0x69e bytes) 000758 1772435328.221960 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0xb762): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb762): Mirror 6: 0x2bf8399 (stale) Mirror 10: 0x2bf8399 (stale) Mirror 13: 0x2bf8399 (stale) Mirror 14: 0x2bf8399 (stale) Mirror 15: 0x2bf8399 (stale) Mirror 17: 0x2bf8399 (stale) Mirror 18: 0x2bf8399 (stale) Mirror 19: 0x2bf8399 (stale) CRC-32 checksum value for chunk [0, 0xb762): Mirror 1: 0xbfe03876 Mirror 20: 0xbfe03876 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb762: skipped 000759 1772435329.410295 mapread 0x008e20 thru 0x00b761 (0x2942 bytes) 000762 1772435329.470775 mapread 0x002f56 thru 0x009216 (0x62c1 bytes) 000764 1772435329.497125 MIRROR_RESYNC 10 mirrors 000766 1772435330.834338 write 0x00522a thru 0x008daf (0x3b86 bytes) truncating to largest ever: 0x3f47d 000770 1772435330.916809 trunc from 0x00b762 to 0x03f47c (0x33d1b bytes) 000772 1772435331.018497 trunc from 0x03f47d to 0x016f90 (0x284ec bytes) 000773 1772435331.057838 read 0x00249f thru 0x011ea1 (0xfa03 bytes) 000774 1772435331.078335 MIRROR_RESYNC 10 mirrors 000775 1772435333.043749 mapread 0x004780 thru 0x011377 (0xcbf8 bytes) 000776 1772435333.119291 MIRROR_RESYNC 10 mirrors 000777 1772435333.617510 write 0x0228ee thru 0x02a08b (0x779e bytes) 000779 1772435333.697324 mapread 0x0287a5 thru 0x02a08b (0x18e7 bytes) 000781 1772435333.699389 mapwrite 0x01c572 thru 0x01cdae (0x83d bytes) 000784 1772435333.824449 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a08c): Mirror 6: 0x55558ace (stale) Mirror 10: 0x55558ace (stale) Mirror 13: 0x55558ace (stale) Mirror 14: 0x55558ace (stale) Mirror 15: 0x55558ace (stale) Mirror 17: 0x55558ace (stale) Mirror 19: 0x55558ace (stale) Mirror 20: 0x55558ace (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a08c): Mirror 6: 0x55558ace (stale) Mirror 10: 0x55558ace (stale) Mirror 13: 0x55558ace (stale) Mirror 14: 0x55558ace (stale) Mirror 15: 0x55558ace (stale) Mirror 17: 0x55558ace (stale) Mirror 19: 0x55558ace (stale) Mirror 20: 0x55558ace (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a08c: skipped 000789 1772435334.517458 mapread 0x02771c thru 0x02a08b (0x2970 bytes) 000792 1772435334.542890 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a08c): Mirror 6: 0x55558ace (stale) Mirror 10: 0x55558ace (stale) Mirror 13: 0x55558ace (stale) Mirror 15: 0x55558ace (stale) Mirror 17: 0x55558ace (stale) Mirror 19: 0x55558ace (stale) Mirror 20: 0x55558ace (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a08c): Mirror 6: 0x55558ace (stale) Mirror 10: 0x55558ace (stale) Mirror 13: 0x55558ace (stale) Mirror 15: 0x55558ace (stale) Mirror 17: 0x55558ace (stale) Mirror 19: 0x55558ace (stale) Mirror 20: 0x55558ace (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a08c: skipped 000793 1772435335.086229 trunc from 0x02a08c to 0x01017e (0x19f0d bytes) 000797 1772435335.130536 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1017f): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1017f): Mirror 6: 0x2c178d3 (stale) Mirror 10: 0x2c178d3 (stale) Mirror 13: 0x2c178d3 (stale) Mirror 15: 0x2c178d3 (stale) Mirror 17: 0x2c178d3 (stale) Mirror 19: 0x2c178d3 (stale) Mirror 20: 0x2c178d3 (stale) CRC-32 checksum value for chunk [0, 0x1017f): Mirror 1: 0x2c178d3 Mirror 21: 0x2c178d3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1017f: skipped 000799 1772435336.673377 MIRROR_RESYNC 9 mirrors 000800 1772435338.418211 mapwrite 0x027c4d thru 0x02f661 (0x7a15 bytes) 000801 1772435338.608099 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f662): Mirror 6: 0x3a7cf6fc (stale) Mirror 10: 0x3a7cf6fc (stale) Mirror 13: 0x3a7cf6fc (stale) Mirror 17: 0x3a7cf6fc (stale) Mirror 19: 0x3a7cf6fc (stale) Mirror 20: 0x3a7cf6fc (stale) Mirror 21: 0x3a7cf6fc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f662): Mirror 6: 0x3a7cf6fc (stale) Mirror 10: 0x3a7cf6fc (stale) Mirror 13: 0x3a7cf6fc (stale) Mirror 17: 0x3a7cf6fc (stale) Mirror 19: 0x3a7cf6fc (stale) Mirror 20: 0x3a7cf6fc (stale) Mirror 21: 0x3a7cf6fc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f662: skipped 000802 1772435339.428168 mapwrite 0x01b9a6 thru 0x0250df (0x973a bytes) 000803 1772435339.535288 mapread 0x01e9c7 thru 0x026196 (0x77d0 bytes) 000807 1772435339.561142 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2f662): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f662): Mirror 6: 0x3a7cf6fc (stale) Mirror 10: 0x3a7cf6fc (stale) Mirror 13: 0x3a7cf6fc (stale) Mirror 17: 0x3a7cf6fc (stale) Mirror 19: 0x3a7cf6fc (stale) Mirror 20: 0x3a7cf6fc (stale) Mirror 21: 0x3a7cf6fc (stale) CRC-32 checksum value for chunk [0, 0x2f662): Mirror 1: 0x16480e55 Mirror 22: 0x16480e55 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f662: skipped 000808 1772435340.976895 read 0x004536 thru 0x013912 (0xf3dd bytes) 000811 1772435341.015226 mapwrite 0x03a711 thru 0x03ffff (0x58ef bytes) 000812 1772435341.207008 mapread 0x027dbf thru 0x0290e8 (0x132a bytes) 000815 1772435341.247489 mapread 0x02cd37 thru 0x03c6ae (0xf978 bytes) 000816 1772435341.311744 mapread 0x035d4d thru 0x03ffff (0xa2b3 bytes) 000817 1772435341.312917 read 0x00a3bb thru 0x01283f (0x8485 bytes) 000818 1772435341.334980 trunc from 0x040000 to 0x02b227 (0x14dd8 bytes) 000819 1772435341.366479 read 0x0071d2 thru 0x016121 (0xef50 bytes) 000820 1772435341.367303 mapwrite 0x0072eb thru 0x00d97c (0x6692 bytes) 000821 1772435341.428360 write 0x030b2e thru 0x03c012 (0xb4e5 bytes) 000822 1772435341.435508 MIRROR_RESYNC 9 mirrors 000826 1772435343.254595 mapread 0x020f5b thru 0x02c6ed (0xb793 bytes) 000827 1772435343.351275 write 0x00fd42 thru 0x01f70f (0xf9ce bytes) 000828 1772435343.532395 mapwrite 0x0164bb thru 0x018e47 (0x298d bytes) 000829 1772435343.639366 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3c013): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c013): Mirror 6: 0x2ec402b3 (stale) Mirror 10: 0x2ec402b3 (stale) Mirror 13: 0x2ec402b3 (stale) Mirror 17: 0x2ec402b3 (stale) Mirror 19: 0x2ec402b3 (stale) Mirror 20: 0x2ec402b3 (stale) Mirror 21: 0x2ec402b3 (stale) Mirror 22: 0x2ec402b3 (stale) CRC-32 checksum value for chunk [0, 0x3c013): Mirror 1: 0xdaa12951 Mirror 23: 0xdaa12951 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c013: skipped 000830 1772435345.047435 mapread 0x0182b8 thru 0x01c4ef (0x4238 bytes) 000832 1772435345.162202 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c013): Mirror 6: 0x2ec402b3 (stale) Mirror 13: 0x2ec402b3 (stale) Mirror 17: 0x2ec402b3 (stale) Mirror 19: 0x2ec402b3 (stale) Mirror 20: 0x2ec402b3 (stale) Mirror 21: 0x2ec402b3 (stale) Mirror 22: 0x2ec402b3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c013): Mirror 6: 0x2ec402b3 (stale) Mirror 13: 0x2ec402b3 (stale) Mirror 17: 0x2ec402b3 (stale) Mirror 19: 0x2ec402b3 (stale) Mirror 20: 0x2ec402b3 (stale) Mirror 21: 0x2ec402b3 (stale) Mirror 22: 0x2ec402b3 (stale) CRC-32 checksum value for chunk [0, 0x3c013): Mirror 1: 0xdaa12951 Mirror 23: 0xdaa12951 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c013: skipped 000833 1772435346.463700 read 0x028af1 thru 0x033876 (0xad86 bytes) 000835 1772435346.515840 trunc from 0x03c013 to 0x014cf4 (0x2731e bytes) 000836 1772435346.652376 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x14cf5): Mirror 6: (stale - read error) Mirror 13: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14cf5): Mirror 6: 0x1fac6263 (stale) Mirror 13: 0x1fac6263 (stale) Mirror 17: 0x1fac6263 (stale) Mirror 19: 0x1fac6263 (stale) Mirror 20: 0x1fac6263 (stale) Mirror 21: 0x1fac6263 (stale) Mirror 22: 0x1fac6263 (stale) Mirror 23: 0xa1a47296 (stale) CRC-32 checksum value for chunk [0, 0x14cf5): Mirror 1: 0xa1a47296 Mirror 24: 0xa1a47296 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14cf5: skipped 000837 1772435347.742502 trunc from 0x014cf5 to 0x025cad (0x10fb9 bytes) 000841 1772435347.847637 trunc from 0x025cae to 0x015348 (0x10965 bytes) 000842 1772435347.875991 mapread 0x0034cd thru 0x00dde1 (0xa915 bytes) 000845 1772435347.909252 trunc from 0x015349 to 0x00c441 (0x8f07 bytes) 000846 1772435347.934498 mapread 0x008b2b thru 0x00c441 (0x3917 bytes) 000847 1772435347.939475 trunc from 0x00c442 to 0x0078bb (0x4b86 bytes) 000849 1772435347.960640 trunc from 0x0078bc to 0x031ed2 (0x2a617 bytes) 000851 1772435347.982400 write 0x034527 thru 0x03ffff (0xbad9 bytes) 000853 1772435347.985728 read 0x026d9d thru 0x029435 (0x2699 bytes) 000854 1772435347.993447 trunc from 0x040000 to 0x00ec8c (0x31373 bytes) 000855 1772435348.016142 trunc from 0x00ec8d to 0x026cbf (0x18033 bytes) 000856 1772435348.050611 mapread 0x00f790 thru 0x018a03 (0x9274 bytes) 000857 1772435348.083280 write 0x029763 thru 0x02ac39 (0x14d7 bytes) 000859 1772435348.089504 mapwrite 0x03cae0 thru 0x03ffff (0x3520 bytes) 000860 1772435348.239924 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x59e730 (stale) Mirror 13: 0x59e730 (stale) Mirror 17: 0x59e730 (stale) Mirror 19: 0x59e730 (stale) Mirror 20: 0x59e730 (stale) Mirror 22: 0x59e730 (stale) Mirror 23: 0xb3886532 (stale) Mirror 24: 0x37570888 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x59e730 (stale) Mirror 13: 0x59e730 (stale) Mirror 17: 0x59e730 (stale) Mirror 19: 0x59e730 (stale) Mirror 20: 0x59e730 (stale) Mirror 22: 0x59e730 (stale) Mirror 23: 0xb3886532 (stale) Mirror 24: 0x37570888 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000861 1772435348.752142 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x59e730 (stale) Mirror 13: 0x59e730 (stale) Mirror 17: 0x59e730 (stale) Mirror 19: 0x59e730 (stale) Mirror 20: 0x59e730 (stale) Mirror 22: 0x59e730 (stale) Mirror 24: 0x37570888 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x59e730 (stale) Mirror 13: 0x59e730 (stale) Mirror 17: 0x59e730 (stale) Mirror 19: 0x59e730 (stale) Mirror 20: 0x59e730 (stale) Mirror 22: 0x59e730 (stale) Mirror 24: 0x37570888 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000862 1772435349.213993 mapread 0x01aff9 thru 0x01d702 (0x270a bytes) 000864 1772435349.240977 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 13: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x59e730 (stale) Mirror 13: 0x59e730 (stale) Mirror 17: 0x59e730 (stale) Mirror 19: 0x59e730 (stale) Mirror 20: 0x59e730 (stale) Mirror 22: 0x59e730 (stale) Mirror 24: 0x37570888 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3e840f15 Mirror 25: 0x3e840f15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000868 1772435350.361904 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x59e730 (stale) Mirror 17: 0x59e730 (stale) Mirror 19: 0x59e730 (stale) Mirror 20: 0x59e730 (stale) Mirror 22: 0x59e730 (stale) Mirror 24: 0x37570888 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x59e730 (stale) Mirror 17: 0x59e730 (stale) Mirror 19: 0x59e730 (stale) Mirror 20: 0x59e730 (stale) Mirror 22: 0x59e730 (stale) Mirror 24: 0x37570888 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3e840f15 Mirror 25: 0x3e840f15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000870 1772435352.024943 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x59e730 (stale) Mirror 19: 0x59e730 (stale) Mirror 20: 0x59e730 (stale) Mirror 22: 0x59e730 (stale) Mirror 24: 0x37570888 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x59e730 (stale) Mirror 19: 0x59e730 (stale) Mirror 20: 0x59e730 (stale) Mirror 22: 0x59e730 (stale) Mirror 24: 0x37570888 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3e840f15 Mirror 25: 0x3e840f15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000876 1772435353.483493 mapwrite 0x00e2a7 thru 0x01c1c7 (0xdf21 bytes) 000878 1772435353.839427 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x59e730 (stale) Mirror 19: 0x59e730 (stale) Mirror 20: 0x59e730 (stale) Mirror 22: 0x59e730 (stale) Mirror 24: 0x37570888 (stale) Mirror 25: 0x3e840f15 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4786e47c Mirror 26: 0x4786e47c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000879 1772435355.092373 read 0x01ed27 thru 0x02470c (0x59e6 bytes) 000882 1772435355.134872 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x59e730 (stale) Mirror 19: 0x59e730 (stale) Mirror 20: 0x59e730 (stale) Mirror 22: 0x59e730 (stale) Mirror 24: 0x37570888 (stale) Mirror 25: 0x3e840f15 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4786e47c Mirror 26: 0x4786e47c Mirror 27: 0x4786e47c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000884 1772435356.166319 mapread 0x010509 thru 0x015d3e (0x5836 bytes) 000885 1772435356.208424 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x59e730 (stale) Mirror 19: 0x59e730 (stale) Mirror 20: 0x59e730 (stale) Mirror 22: 0x59e730 (stale) Mirror 25: 0x3e840f15 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x59e730 (stale) Mirror 19: 0x59e730 (stale) Mirror 20: 0x59e730 (stale) Mirror 22: 0x59e730 (stale) Mirror 25: 0x3e840f15 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4786e47c Mirror 26: 0x4786e47c Mirror 27: 0x4786e47c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000886 1772435357.245880 mapread 0x035b59 thru 0x037746 (0x1bee bytes) 000887 1772435357.263175 read 0x0204ca thru 0x0246af (0x41e6 bytes) 000889 1772435357.272482 read 0x00a291 thru 0x00fd1a (0x5a8a bytes) 000892 1772435357.288118 write 0x036166 thru 0x03a46f (0x430a bytes) 000893 1772435357.437889 write 0x0130c0 thru 0x017775 (0x46b6 bytes) 000896 1772435357.458450 read 0x03b8b4 thru 0x03ffff (0x474c bytes) 000897 1772435357.492406 mapread 0x027eca thru 0x028cf3 (0xe2a bytes) 000898 1772435357.498751 trunc from 0x040000 to 0x011f0e (0x2e0f1 bytes) 000899 1772435357.540011 trunc from 0x011f0f to 0x03964a (0x2773c bytes) 000900 1772435357.566543 write 0x01f6bc thru 0x026bc0 (0x7505 bytes) 000901 1772435357.579508 trunc from 0x03964b to 0x03c4ea (0x2ea0 bytes) 000902 1772435357.612110 mapread 0x02a9a2 thru 0x031eaa (0x7509 bytes) 000905 1772435357.678653 read 0x010463 thru 0x01b0d7 (0xac75 bytes) 000907 1772435357.708703 read 0x0089d3 thru 0x00fa8e (0x70bc bytes) 000909 1772435357.713352 read 0x03010f thru 0x03ba32 (0xb924 bytes) 000910 1772435357.715367 mapwrite 0x017fd9 thru 0x01c242 (0x426a bytes) 000911 1772435357.824525 mapwrite 0x01fc01 thru 0x02b132 (0xb532 bytes) 000913 1772435357.953747 write 0x03aad1 thru 0x03ffff (0x552f bytes) 000914 1772435357.962680 mapwrite 0x00144b thru 0x00d09b (0xbc51 bytes) 000915 1772435358.104108 mapread 0x02ce3c thru 0x035e0f (0x8fd4 bytes) 000917 1772435358.109247 mapread 0x03fd6c thru 0x03ffff (0x294 bytes) 000919 1772435358.122230 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x59e730 (stale) Mirror 19: 0x59e730 (stale) Mirror 20: 0x59e730 (stale) Mirror 22: 0x59e730 (stale) Mirror 25: 0x3e840f15 (stale) Mirror 26: 0x4786e47c (stale) Mirror 27: 0x4786e47c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3fa0fb35 Mirror 28: 0x3fa0fb35 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000920 1772435359.540860 read 0x01ebbb thru 0x029aee (0xaf34 bytes) 000922 1772435359.604648 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x59e730 (stale) Mirror 19: 0x59e730 (stale) Mirror 20: 0x59e730 (stale) Mirror 22: 0x59e730 (stale) Mirror 25: 0x3e840f15 (stale) Mirror 27: 0x4786e47c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x59e730 (stale) Mirror 19: 0x59e730 (stale) Mirror 20: 0x59e730 (stale) Mirror 22: 0x59e730 (stale) Mirror 25: 0x3e840f15 (stale) Mirror 27: 0x4786e47c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3fa0fb35 Mirror 28: 0x3fa0fb35 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000923 1772435360.599483 write 0x0306b1 thru 0x0379f3 (0x7343 bytes) 000924 1772435360.763167 trunc from 0x040000 to 0x01c907 (0x236f8 bytes) 000926 1772435360.820348 mapwrite 0x01b2f5 thru 0x029ee2 (0xebee bytes) 000928 1772435360.978272 mapwrite 0x037456 thru 0x03ffff (0x8baa bytes) 000929 1772435361.148664 mapread 0x033f33 thru 0x0389dd (0x4aab bytes) 000930 1772435361.152313 mapwrite 0x005858 thru 0x00e954 (0x90fd bytes) 000932 1772435361.292437 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x59e730 (stale) Mirror 19: 0x59e730 (stale) Mirror 20: 0x59e730 (stale) Mirror 22: 0x59e730 (stale) Mirror 25: 0x3e840f15 (stale) Mirror 28: 0x3fa0fb35 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x59e730 (stale) Mirror 19: 0x59e730 (stale) Mirror 20: 0x59e730 (stale) Mirror 22: 0x59e730 (stale) Mirror 25: 0x3e840f15 (stale) Mirror 28: 0x3fa0fb35 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000933 1772435361.820886 mapwrite 0x015ae0 thru 0x019fda (0x44fb bytes) 000935 1772435361.900069 mapwrite 0x020c50 thru 0x0306ab (0xfa5c bytes) 000937 1772435362.039619 write 0x013c22 thru 0x0148ab (0xc8a bytes) 000938 1772435362.040707 mapread 0x02c4a0 thru 0x035de2 (0x9943 bytes) 000939 1772435362.060248 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x59e730 (stale) Mirror 19: 0x59e730 (stale) Mirror 20: 0x59e730 (stale) Mirror 22: 0x59e730 (stale) Mirror 25: 0x3e840f15 (stale) Mirror 28: 0x3fa0fb35 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8f04546a Mirror 29: 0x8f04546a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000940 1772435363.057945 mapwrite 0x006c3d thru 0x0154b8 (0xe87c bytes) 000941 1772435363.362523 mapread 0x015b5c thru 0x025ae0 (0xff85 bytes) 000942 1772435363.383789 mapwrite 0x013c27 thru 0x0199c6 (0x5da0 bytes) 000945 1772435363.447789 write 0x01aae1 thru 0x01b0c4 (0x5e4 bytes) 000946 1772435363.450448 write 0x0095dd thru 0x010552 (0x6f76 bytes) 000948 1772435363.459295 MIRROR_RESYNC 8 mirrors 000949 1772435365.358022 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) 000950 1772435365.966735 trunc from 0x040000 to 0x00f7c9 (0x30836 bytes) 000953 1772435366.092768 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf7ca): Mirror 6: 0xf3d7eda7 (stale) Mirror 19: 0xf3d7eda7 (stale) Mirror 20: 0xf3d7eda7 (stale) Mirror 25: 0xf3d7eda7 (stale) Mirror 29: 0xf3d7eda7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf7ca): Mirror 6: 0xf3d7eda7 (stale) Mirror 19: 0xf3d7eda7 (stale) Mirror 20: 0xf3d7eda7 (stale) Mirror 25: 0xf3d7eda7 (stale) Mirror 29: 0xf3d7eda7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf7ca: skipped 000956 1772435366.600616 mapread 0x0041e7 thru 0x0062e5 (0x20ff bytes) 000957 1772435366.634211 read 0x00f69a thru 0x00f7c9 (0x130 bytes) 000959 1772435366.638029 mapread 0x0067bc thru 0x00f6b7 (0x8efc bytes) 000960 1772435366.640198 trunc from 0x00f7ca to 0x01fbed (0x10424 bytes) 000971 1772435366.671873 trunc from 0x01fbee to 0x014df0 (0xadfd bytes) 000972 1772435366.684787 mapwrite 0x034be2 thru 0x037371 (0x2790 bytes) 000973 1772435366.771021 read 0x00334f thru 0x00bd01 (0x89b3 bytes) 000975 1772435366.774732 read 0x02cbdc thru 0x036037 (0x945c bytes) 000976 1772435366.786065 mapwrite 0x03ef54 thru 0x03ffff (0x10ac bytes) 000977 1772435366.865548 mapwrite 0x005e15 thru 0x0156fd (0xf8e9 bytes) 000978 1772435366.970932 mapwrite 0x00611e thru 0x0139a7 (0xd88a bytes) 000980 1772435367.084903 read 0x02a70a thru 0x031243 (0x6b3a bytes) 000981 1772435367.085738 mapwrite 0x02bcd7 thru 0x03a7f6 (0xeb20 bytes) 000983 1772435367.177216 write 0x032e4a thru 0x03b7fe (0x89b5 bytes) 000984 1772435367.185950 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 25: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x23aa4990 (stale) Mirror 19: 0x23aa4990 (stale) Mirror 20: 0x23aa4990 (stale) Mirror 25: 0x23aa4990 (stale) Mirror 29: 0x23aa4990 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5a3fde5c Mirror 30: 0x5a3fde5c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000989 1772435368.683954 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 25: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x23aa4990 (stale) Mirror 19: 0x23aa4990 (stale) Mirror 20: 0x23aa4990 (stale) Mirror 25: 0x23aa4990 (stale) Mirror 29: 0x23aa4990 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5a3fde5c Mirror 30: 0x5a3fde5c Mirror 31: 0x5a3fde5c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000992 1772435369.917944 mapwrite 0x027cba thru 0x02dc6c (0x5fb3 bytes) 000993 1772435370.220549 mapread 0x031e0a thru 0x03c98c (0xab83 bytes) 000994 1772435370.261700 read 0x02b512 thru 0x02db66 (0x2655 bytes) 000997 1772435370.265095 mapread 0x03284b thru 0x03aabc (0x8272 bytes) 000999 1772435370.267351 mapwrite 0x00e583 thru 0x01b261 (0xccdf bytes) All operations completed A-OK! PASS 70a (124s) == sanity-flr test 71: check mirror extend parent fid ==== 02:09:35 (1772435375) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 02:09:39 (1772435379) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x251:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x24d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.369035 s, 5.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x251:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x252:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x24d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20174 s, 5.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x251:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x252:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x24e:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x24d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x251:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x252:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x24e:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x24d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x253:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 02:09:49 (1772435389) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre2 Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6939264 .. Extending file size to 3536384 .. failed Extending file size to 6954048 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8059104 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7871296 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8137152 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8963712 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9371776 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8711424 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8823904 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 55442 55443 55445 55446 55448 done 192.168.203.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg316-client.virtnet /mnt/lustre2 (opts:) 192.168.203.116@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg316-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 02:11:09 (1772435469) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre2 Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5150560 .. done Extending file size to 3735808 .. Extending file size to 5344320 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9200640 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6204896 .. Extending file size to 9223520 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9331520 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57332 57333 57334 57335 57339 done 192.168.203.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg316-client.virtnet /mnt/lustre2 (opts:) 192.168.203.116@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg316-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 02:12:29 (1772435549) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 25600 3743744 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 25600 3743744 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123263 s, 17.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.461519 s, 4.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xe56c58e1 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg316-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 02:12:50 (1772435570) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.10172 s, 20.6 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 02:12:58 (1772435578) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f1:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x264:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x264:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x265:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x268:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 02:13:06 (1772435586) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.22812 s, 9.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x266:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x269:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 02:13:13 (1772435593) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.328557 s, 6.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x267:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x268:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x267:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x268:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26c:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 02:13:23 (1772435603) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.269126 s, 7.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x269:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x269:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26e:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 02:13:57 (1772435637) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg316-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110584 s, 9.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x270:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x270:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26b:0x0] } PASS 204d (59s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 02:14:56 (1772435696) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg316-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.415888 s, 5.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x271:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x271:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (61s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 02:15:58 (1772435758) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg316-server mds-ost sync done. Creating new pool test_204f oleg316-server: Pool lustre.test_204f created Adding targets to pool oleg316-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg316-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.453571 s, 4.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x272:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x272:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg316-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg316-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg316-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 02:17:19 (1772435839) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 02:17:27 (1772435847) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 02:17:35 (1772435855) oleg316-server: Pool lustre.test_206 created Adding targets to pool oleg316-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg316-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg316-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg316-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg316-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 02:18:00 (1772435880) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519344 s, 20.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666963 s, 15.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xe40379c9 Mirror 3: 0xe40379c9 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 02:18:10 (1772435890) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 02:18:14 (1772435894) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 02:18:18 (1772435898) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.152345 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1010000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2590000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2590000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2590000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2500000 Read this (4 bytes): BBBB 192.168.203.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg316-client.virtnet /mnt/lustre2 (opts:) PASS 209a (34s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 02:18:52 (1772435932) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0161103 s, 508 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 02:18:59 (1772435939) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461 s, 22.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x216 lmm_fid: [0x200000404:0x216:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x216 lmm_fid: [0x200000404:0x216:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27d:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834454 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-02 02:19:01.000000000 -0500 Modify: 2026-03-02 02:19:01.000000000 -0500 Change: 2026-03-02 02:19:02.000000000 -0500 Birth: 2026-03-02 02:19:01.000000000 -0500 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 02:19:07 (1772435947) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 02:19:10 (1772435950) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.720519 s, 56.8 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0589268 s, 69.5 kB/s size after second write -rw-r--r-- 1 root root 4096 Mar 2 02:19 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x218:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Mar 2 02:19 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 02:19:19 (1772435959) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 02:19:26 (1772435966) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1694 sec =========== 02:19:34 (1772435974) === sanity-flr: start cleanup 02:19:36 (1772435976) === === sanity-flr: finish cleanup 02:19:42 (1772435982) ===