-----============= acceptance-small: sanity-flr ============----- Thu Jan 1 19:57:40 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg430-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 19:58:01 (1767315481) === oleg430-client.virtnet: executing check_config_client /mnt/lustre oleg430-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg430-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff927518c0f800.idle_timeout=debug osc.lustre-OST0001-osc-ffff927518c0f800.idle_timeout=debug disable quota as required oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 19:58:57 (1767315537) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f9334] == sanity-flr test 0a: lfs mirror create with -N option == 19:58:59 (1767315539) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 19:59:18 (1767315558) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (5s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 19:59:24 (1767315564) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (6s) == sanity-flr test 0d: lfs mirror extend with -N option == 19:59:30 (1767315570) PASS 0d (35s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:00:05 (1767315605) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (5s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:00:10 (1767315610) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 20:00:15 (1767315615) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff927518c0f800.stats=clear osc.lustre-OST0001-osc-ffff927518c0f800.stats=clear PASS 0g (11s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:00:25 (1767315625) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (15s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:00:40 (1767315640) PASS 0j (12s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:00:52 (1767315652) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:00:55 (1767315655) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (12s) == sanity-flr test 2: create components from existing files ========================================================== 20:01:07 (1767315667) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:01:15 (1767315675) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:01:18 (1767315678) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:01:27 (1767315687) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:01:39 (1767315699) PASS 7 (9s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:01:49 (1767315709) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 1.28775 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.423766 s, 2.5 MB/s PASS 21 (14s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:02:03 (1767315723) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 0.537326 s, 35.1 MB/s PASS 22 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:02:13 (1767315733) Stopping /mnt/lustre-ost1 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff927518c0f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff927518c0f800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg430-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff927518c0f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff927518c0f800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg430-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (95s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:03:48 (1767315828) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.539173 s, 13.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff927518c0f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff927518c0f800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg430-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (58s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:04:46 (1767315886) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff927518c0f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff927518c0f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg430-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff927518c0f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff927518c0f800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg430-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (91s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:06:18 (1767315978) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 7.04288 s, 7.4 MB/s -rw-r--r-- 1 root root 52428800 Jan 1 20:06 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.551s user 0m0.001s sys 0m0.179s Stopping /mnt/lustre-ost1 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff927518c0f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff927518c0f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.598s user 0m0.000s sys 0m0.207s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg430-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff927518c0f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff927518c0f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.355s user 0m0.000s sys 0m0.168s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg430-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg430-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff927518c0f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff927518c0f800.ost_server_uuid in FULL state after 0 sec PASS 33b (105s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:08:02 (1767316082) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:08:06 (1767316086) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:08:09 (1767316089) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 20:08:12 (1767316092) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 20:08:19 (1767316099) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.469049 s, 26.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:08:32 (1767316112) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318443 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497402 s, 21.1 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:08:39 (1767316119) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:08:48 (1767316128) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0016566 s, 618 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00299112 s, 342 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00104663 s, 978 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00200794 s, 510 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 20:09:00 (1767316140) create test files with size 15733807 .. 129611c7b6b4ab3c03958dba58fb82bb - b8198884dd82ac2548894506ac41e34e - 175c786181947a3ca224674b239e688e - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg430-client.virtnet /mnt/lustre (opts:) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre PASS 37 (28s) == sanity-flr test 38: resync ============================ 20:09:28 (1767316168) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:09:46 (1767316186) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:09:54 (1767316194) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109555 s, 9.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276678 s, 37.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 20:10:04 (1767316204) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.147028 s, 14.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.332003 s, 12.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.307707 s, 6.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 20:10:24 (1767316224) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 20:10:28 (1767316228) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:10:32 (1767316232) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238712 s, 43.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 20:10:41 (1767316241) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:10:44 (1767316244) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:10:54 (1767316254) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.358142 s, 29.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg430-client.virtnet /mnt/lustre (opts:) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:11:06 (1767316266) llite.lustre-ffff927539ec9000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.296999 s, 35.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg430-client.virtnet /mnt/lustre (opts:) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff927503727000.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:11:21 (1767316281) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (42s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:12:03 (1767316323) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (37s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:12:41 (1767316361) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 20:12:45 (1767316365) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.220333 s, 14.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.185542 s, 17.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xaa9fe9d1 (stale) Mirror 2: 0xaa9fe9d1 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: eb6c07f1e8827142723dd3af938cc639 - mirror 1 checksum: b526852cbba7185526220b902050afc4 - mirror 2 checksum: eb6c07f1e8827142723dd3af938cc639 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jan 1 20:12:49 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jan 1 20:12:49 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: eb6c07f1e8827142723dd3af938cc639 - mirror 1 checksum: b526852cbba7185526220b902050afc4 - mirror 2 checksum: b526852cbba7185526220b902050afc4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jan 1 20:12:49 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jan 1 20:12:49 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 48 (13s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:13:00 (1767316380) Failing mds1 on oleg430-server Stopping /mnt/lustre-mds1 (opts:) on oleg430-server 20:13:08 (1767316388) shut down facet: mds1 facet_host: oleg430-server facet_failover_host: oleg430-server Failover mds1 to oleg430-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-MDT0000 20:13:41 (1767316421) targets are mounted 20:13:41 (1767316421) facet_failover done oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305513 s, 34.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b:0x0] } PASS 50A (57s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:13:58 (1767316438) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:14:01 (1767316441) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (5s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:14:06 (1767316446) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0101236 s, 1.6 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:14:15 (1767316455) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:14:19 (1767316459) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:14:24 (1767316464) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:14:48 (1767316488) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:15:17 (1767316517) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:15:40 (1767316540) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:16:05 (1767316565) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 20:16:12 (1767316572) mirror create pid 54795 mirror split pid 54796 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7808: 54795 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7808: 54796 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 20:17:19 (1767316639) Seed set to 6640 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x2a9f3 000004 1767316641.259330 trunc from 00000000 to 0x02a9f2 (0x2a9f3 bytes) 000006 1767316641.279459 write 0x0071a5 thru 0x00ade6 (0x3c42 bytes) 000007 1767316641.294461 read 0x01fd32 thru 0x02a9f2 (0xacc1 bytes) 000008 1767316641.306379 mapwrite 0x0011cf thru 0x005aff (0x4931 bytes) 000009 1767316641.425409 trunc from 0x02a9f3 to 0x0051c3 (0x2582f bytes) 000011 1767316641.440110 mapwrite 0x03e004 thru 0x03ffff (0x1ffc bytes) 000013 1767316641.536392 read 0x010406 thru 0x020131 (0xfd2c bytes) 000015 1767316641.553779 mapwrite 0x03076b thru 0x03ff46 (0xf7dc bytes) 000019 1767316641.633150 mapread 0x03d49c thru 0x03ffff (0x2b64 bytes) 000020 1767316641.634593 trunc from 0x040000 to 0x01deec (0x22113 bytes) 000025 1767316641.654517 trunc from 0x01deed to 0x0168d0 (0x761c bytes) 000026 1767316641.675437 read 0x003e27 thru 0x00958f (0x5769 bytes) 000029 1767316641.677203 write 0x01cfab thru 0x02c046 (0xf09c bytes) 000030 1767316641.680346 write 0x00d58f thru 0x0145c3 (0x7035 bytes) 000033 1767316641.683443 write 0x00567e thru 0x00851d (0x2ea0 bytes) 000034 1767316641.685076 read 0x0287be thru 0x02a6ac (0x1eef bytes) 000036 1767316641.686592 read 0x0234eb thru 0x02c046 (0x8b5c bytes) 000037 1767316641.688004 write 0x03345f thru 0x03ffff (0xcba1 bytes) 000038 1767316641.689757 read 0x03af88 thru 0x03ffff (0x5078 bytes) 000041 1767316641.693455 read 0x01fb81 thru 0x027632 (0x7ab2 bytes) 000046 1767316641.697110 mapread 0x016588 thru 0x017dc7 (0x1840 bytes) 000047 1767316641.706125 mapwrite 0x01f9e9 thru 0x02a1b6 (0xa7ce bytes) truncating to largest ever: 0x366f7 000050 1767316641.758928 trunc from 0x040000 to 0x0366f6 (0x9909 bytes) 000051 1767316641.773955 mapwrite 0x01b0dd thru 0x022f58 (0x7e7c bytes) 000052 1767316641.814751 mapread 0x00a181 thru 0x00d6cc (0x354c bytes) 000053 1767316641.815456 read 0x018fb9 thru 0x01bb1c (0x2b64 bytes) truncating to largest ever: 0x37342 000055 1767316641.816270 trunc from 0x0366f7 to 0x037341 (0xc4b bytes) 000057 1767316641.828532 mapread 0x02d1c5 thru 0x02e55c (0x1398 bytes) 000058 1767316641.832370 MIRROR_EXTEND to 2 mirrors 000060 1767316642.602328 write 0x0239a5 thru 0x027a49 (0x40a5 bytes) 000061 1767316642.682351 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37342): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37342): Mirror 2: 0xedf135bf (stale) CRC-32 checksum value for chunk [0, 0x37342): Mirror 1: 0xeb6c0221 Mirror 3: 0xeb6c0221 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37342: skipped 000063 1767316643.564257 mapread 0x018985 thru 0x019c16 (0x1292 bytes) 000065 1767316643.606189 mapread 0x01701c thru 0x02508f (0xe074 bytes) 000066 1767316643.639294 write 0x021b60 thru 0x02222d (0x6ce bytes) 000067 1767316643.715168 trunc from 0x037342 to 0x00ffbc (0x27385 bytes) 000068 1767316643.733278 mapwrite 0x00bede thru 0x0125b9 (0x66dc bytes) 000069 1767316643.805399 mapwrite 0x025b53 thru 0x030f59 (0xb407 bytes) 000073 1767316643.884444 write 0x011aef thru 0x018c6a (0x717c bytes) 000074 1767316643.891992 mapread 0x0046a3 thru 0x0095e5 (0x4f43 bytes) 000075 1767316643.897247 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x30f5a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30f5a): Mirror 2: 0xc7d1f973 (stale) Mirror 3: 0xd7a1a823 (stale) CRC-32 checksum value for chunk [0, 0x30f5a): Mirror 1: 0x6f85279b Mirror 4: 0x6f85279b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30f5a: skipped 000076 1767316644.873917 write 0x013b02 thru 0x01e25b (0xa75a bytes) 000078 1767316644.962333 trunc from 0x030f5a to 0x02ffc8 (0xf91 bytes) 000079 1767316644.992751 trunc from 0x02ffc9 to 0x01a74d (0x1587b bytes) 000080 1767316645.013910 mapwrite 0x002982 thru 0x00a273 (0x78f2 bytes) 000081 1767316645.093011 mapread 0x00de52 thru 0x0136e5 (0x5894 bytes) 000082 1767316645.100456 mapwrite 0x037e1d thru 0x03b5f0 (0x37d4 bytes) 000083 1767316645.188701 read 0x03427b thru 0x03b5f0 (0x7376 bytes) 000084 1767316645.199184 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3b5f1): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b5f1): Mirror 2: 0x5794ca2a (stale) Mirror 3: 0x4c764cc5 (stale) Mirror 4: 0xa0005d17 (stale) CRC-32 checksum value for chunk [0, 0x3b5f1): Mirror 1: 0xe571c1a9 Mirror 5: 0xe571c1a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b5f1: skipped 000085 1767316646.259230 write 0x028816 thru 0x02df27 (0x5712 bytes) 000087 1767316646.375677 write 0x025bf1 thru 0x02c1ed (0x65fd bytes) 000090 1767316646.396275 trunc from 0x03b5f1 to 0x02d9f0 (0xdc00 bytes) 000091 1767316646.521062 write 0x0037f6 thru 0x00e0df (0xa8ea bytes) 000093 1767316646.538309 mapread 0x01260c thru 0x01669c (0x4091 bytes) 000094 1767316646.557343 mapwrite 0x0094e8 thru 0x0118e7 (0x8400 bytes) 000096 1767316646.597289 mapwrite 0x032b86 thru 0x036bc9 (0x4044 bytes) 000098 1767316646.720777 read 0x00b46c thru 0x017e46 (0xc9db bytes) 000099 1767316646.721494 mapwrite 0x02e504 thru 0x02fced (0x17ea bytes) 000101 1767316646.825364 mapwrite 0x02d0a9 thru 0x02fcb4 (0x2c0c bytes) 000104 1767316646.983897 read 0x01761e thru 0x01bba2 (0x4585 bytes) 000108 1767316647.020439 write 0x00cbf3 thru 0x017b74 (0xaf82 bytes) 000109 1767316647.028444 mapwrite 0x03d9b2 thru 0x03ffff (0x264e bytes) 000110 1767316647.162373 mapwrite 0x01604b thru 0x023993 (0xd949 bytes) 000111 1767316647.243209 read 0x02056b thru 0x02a44f (0x9ee5 bytes) 000113 1767316647.244433 write 0x030dbd thru 0x03d67b (0xc8bf bytes) 000115 1767316647.249048 trunc from 0x040000 to 0x02b7c6 (0x14839 bytes) 000116 1767316647.273383 write 0x01cc17 thru 0x028c95 (0xc07f bytes) 000118 1767316647.276673 mapwrite 0x00bee6 thru 0x017eb0 (0xbfcb bytes) 000119 1767316647.353231 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b7c7): Mirror 2: 0x2b72dc74 (stale) Mirror 4: 0x925f0def (stale) Mirror 5: 0x251f4044 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b7c7): Mirror 2: 0x2b72dc74 (stale) Mirror 4: 0x925f0def (stale) Mirror 5: 0x251f4044 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b7c7: skipped 000122 1767316647.763150 mapwrite 0x000cac thru 0x00b5ac (0xa901 bytes) 000123 1767316647.843193 mapwrite 0x0223c4 thru 0x024fb7 (0x2bf4 bytes) 000124 1767316647.891034 trunc from 0x02b7c7 to 0x028b44 (0x2c82 bytes) 000127 1767316647.911072 read 0x00f98b thru 0x01ebf3 (0xf269 bytes) 000128 1767316647.918572 mapwrite 0x000513 thru 0x00ecd9 (0xe7c7 bytes) 000129 1767316648.053110 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x28b45): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28b45): Mirror 2: 0x2611da8c (stale) Mirror 4: 0xe7d7af41 (stale) Mirror 5: 0xa75a454b (stale) CRC-32 checksum value for chunk [0, 0x28b45): Mirror 1: 0x7adcea54 Mirror 6: 0x7adcea54 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28b45: skipped 000131 1767316649.343264 trunc from 0x028b45 to 0x0341d8 (0xb694 bytes) 000132 1767316649.463807 read 0x00befb thru 0x00e389 (0x248f bytes) 000133 1767316649.475431 write 0x006284 thru 0x012891 (0xc60e bytes) 000134 1767316649.503279 MIRROR_RESYNC 5 mirrors 000135 1767316650.753628 mapwrite 0x01bd7c thru 0x02128d (0x5512 bytes) 000136 1767316651.109370 mapwrite 0x02f49b thru 0x03040e (0xf74 bytes) 000138 1767316651.240463 read 0x011eca thru 0x01d409 (0xb540 bytes) 000139 1767316651.254245 mapread 0x01d073 thru 0x01ec64 (0x1bf2 bytes) 000140 1767316651.266572 MIRROR_RESYNC 5 mirrors 000141 1767316652.344482 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) 000142 1767316652.648369 write 0x00f9bd thru 0x01cda3 (0xd3e7 bytes) 000146 1767316652.753926 trunc from 0x0341d9 to 0x00e099 (0x2613f bytes) 000147 1767316652.797220 mapwrite 0x022a04 thru 0x025e49 (0x3446 bytes) 000148 1767316652.887548 mapread 0x005901 thru 0x013f9f (0xe69f bytes) 000151 1767316652.911623 read 0x021144 thru 0x025e49 (0x4d06 bytes) 000152 1767316652.912234 read 0x01254a thru 0x017ad0 (0x5587 bytes) 000153 1767316652.912764 write 0x01ce02 thru 0x02751c (0xa71b bytes) 000155 1767316652.916171 write 0x005e36 thru 0x00fe5a (0xa025 bytes) 000156 1767316652.918102 write 0x004329 thru 0x00dcd3 (0x99ab bytes) 000157 1767316652.920569 mapread 0x01cf31 thru 0x02751c (0xa5ec bytes) 000158 1767316652.921952 mapwrite 0x02fbf4 thru 0x0300e9 (0x4f6 bytes) 000160 1767316653.059323 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x300ea): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x300ea): Mirror 2: 0xe1b1bf21 (stale) Mirror 4: 0xe1b1bf21 (stale) Mirror 5: 0xe1b1bf21 (stale) CRC-32 checksum value for chunk [0, 0x300ea): Mirror 1: 0x18f2654 Mirror 6: 0x18f2654 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x300ea: skipped 000161 1767316654.038906 MIRROR_RESYNC 5 mirrors 000162 1767316655.009353 read 0x015c8f thru 0x01ef39 (0x92ab bytes) 000163 1767316655.019056 trunc from 0x0300ea to 0x03110e (0x1025 bytes) 000164 1767316655.105588 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3110f): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3110f): Mirror 2: 0xdaa606c8 (stale) Mirror 4: 0xdaa606c8 (stale) Mirror 5: 0xdaa606c8 (stale) Mirror 6: 0xdaa606c8 (stale) CRC-32 checksum value for chunk [0, 0x3110f): Mirror 1: 0xdaa606c8 Mirror 7: 0xdaa606c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3110f: skipped truncating to largest ever: 0x3bb4f 000165 1767316656.001464 trunc from 0x03110f to 0x03bb4e (0xaa40 bytes) 000166 1767316656.102766 write 0x018196 thru 0x02264d (0xa4b8 bytes) 000167 1767316656.117140 mapread 0x002479 thru 0x003d67 (0x18ef bytes) 000168 1767316656.128103 trunc from 0x03bb4f to 0x032db3 (0x8d9b bytes) 000169 1767316656.148060 mapread 0x01a0d9 thru 0x02279c (0x86c4 bytes) 000170 1767316656.162377 mapwrite 0x01da63 thru 0x027d33 (0xa2d1 bytes) 000171 1767316656.209236 mapread 0x030cac thru 0x032db3 (0x2108 bytes) 000174 1767316656.216801 mapread 0x02e894 thru 0x02f4a0 (0xc0d bytes) 000176 1767316656.217917 mapread 0x014872 thru 0x017abc (0x324b bytes) 000178 1767316656.225758 read 0x02d257 thru 0x032db3 (0x5b5d bytes) 000179 1767316656.226475 mapwrite 0x02df58 thru 0x02f388 (0x1431 bytes) 000182 1767316656.277684 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32db4): Mirror 2: 0xd084e1d6 (stale) Mirror 4: 0xd084e1d6 (stale) Mirror 5: 0xd084e1d6 (stale) Mirror 7: 0xd084e1d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32db4): Mirror 2: 0xd084e1d6 (stale) Mirror 4: 0xd084e1d6 (stale) Mirror 5: 0xd084e1d6 (stale) Mirror 7: 0xd084e1d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32db4: skipped 000184 1767316656.706932 read 0x007330 thru 0x013927 (0xc5f8 bytes) 000185 1767316656.719797 read 0x01754d thru 0x022b2f (0xb5e3 bytes) 000187 1767316656.732741 mapwrite 0x02f261 thru 0x03b8d2 (0xc672 bytes) 000189 1767316656.897973 trunc from 0x03b8d3 to 0x03984f (0x2083 bytes) 000190 1767316656.927926 write 0x02fe14 thru 0x03acd2 (0xaebf bytes) 000191 1767316656.929474 mapread 0x00f82e thru 0x01801c (0x87ef bytes) 000192 1767316656.931459 mapread 0x013e40 thru 0x01e88d (0xaa4e bytes) 000193 1767316656.932969 write 0x0303af thru 0x03f08e (0xece0 bytes) 000194 1767316656.935207 trunc from 0x03f08f to 0x003271 (0x3be1d bytes) 000195 1767316656.954419 read 0x001ed0 thru 0x003271 (0x13a2 bytes) truncating to largest ever: 0x3c8be 000196 1767316656.955654 trunc from 0x003272 to 0x03c8bd (0x3964c bytes) 000197 1767316656.978439 mapread 0x0126a8 thru 0x01e1d3 (0xbb2c bytes) 000198 1767316656.991159 trunc from 0x03c8be to 0x00b84c (0x31071 bytes) 000199 1767316657.006124 read 0x001826 thru 0x00a6c2 (0x8e9d bytes) 000200 1767316657.006695 read 0x009ffd thru 0x00ae47 (0xe4b bytes) 000201 1767316657.007900 trunc from 0x00b84d to 0x009820 (0x202c bytes) 000202 1767316657.023021 write 0x00e41d thru 0x0189c7 (0xa5ab bytes) 000203 1767316657.024273 write 0x027b1d thru 0x02a48a (0x296e bytes) 000206 1767316657.029818 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a48b): Mirror 2: 0xf3f8dc83 (stale) Mirror 4: 0xf3f8dc83 (stale) Mirror 5: 0xf3f8dc83 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a48b): Mirror 2: 0xf3f8dc83 (stale) Mirror 4: 0xf3f8dc83 (stale) Mirror 5: 0xf3f8dc83 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a48b: skipped 000209 1767316657.388484 trunc from 0x02a48b to 0x01654d (0x13f3d bytes) 000212 1767316657.413666 trunc from 0x01654e to 0x024843 (0xe2f6 bytes) 000213 1767316657.441774 mapread 0x00fc1e thru 0x01fbab (0xff8e bytes) 000214 1767316657.486435 mapread 0x024673 thru 0x024843 (0x1d1 bytes) 000215 1767316657.493660 write 0x0275ba thru 0x034da5 (0xd7ec bytes) 000216 1767316657.506827 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x34da6): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34da6): Mirror 2: 0x1cf9a04d (stale) Mirror 4: 0x1cf9a04d (stale) Mirror 5: 0x1cf9a04d (stale) CRC-32 checksum value for chunk [0, 0x34da6): Mirror 1: 0xcee2c8e7 Mirror 6: 0xcee2c8e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34da6: skipped 000219 1767316658.324717 mapread 0x0284a6 thru 0x030447 (0x7fa2 bytes) 000224 1767316658.370434 read 0x00ecc6 thru 0x01016f (0x14aa bytes) 000226 1767316658.385653 trunc from 0x034da6 to 0x01d046 (0x17d5f bytes) 000228 1767316658.483602 mapwrite 0x00ffec thru 0x0146f7 (0x470c bytes) 000229 1767316658.588258 mapwrite 0x03b439 thru 0x03ffff (0x4bc7 bytes) 000230 1767316658.688230 mapread 0x02b2ef thru 0x02b37c (0x8e bytes) 000231 1767316658.703199 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xdfe7e751 (stale) Mirror 5: 0xdfe7e751 (stale) Mirror 6: 0x38076813 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xdfe7e751 (stale) Mirror 5: 0xdfe7e751 (stale) Mirror 6: 0x38076813 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000232 1767316659.125440 mapwrite 0x02b619 thru 0x037abd (0xc4a5 bytes) 000233 1767316659.207929 mapread 0x028b50 thru 0x033932 (0xade3 bytes) 000234 1767316659.209559 trunc from 0x040000 to 0x034482 (0xbb7d bytes) 000238 1767316659.247838 write 0x029f11 thru 0x033d42 (0x9e32 bytes) 000239 1767316659.250159 mapread 0x01bdfd thru 0x01d786 (0x198a bytes) 000241 1767316659.293980 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34483): Mirror 2: 0x5eab2ab6 (stale) Mirror 5: 0x5eab2ab6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34483): Mirror 2: 0x5eab2ab6 (stale) Mirror 5: 0x5eab2ab6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34483: skipped 000242 1767316659.611989 mapwrite 0x03a448 thru 0x03ffff (0x5bb8 bytes) 000243 1767316659.724376 read 0x028e02 thru 0x02d71f (0x491e bytes) 000248 1767316659.743904 MIRROR_RESYNC 3 mirrors 000249 1767316660.453149 read 0x022464 thru 0x026b3b (0x46d8 bytes) 000250 1767316660.466230 read 0x035ba4 thru 0x03c790 (0x6bed bytes) 000251 1767316660.487373 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) 000252 1767316660.809826 mapread 0x00d528 thru 0x011f5d (0x4a36 bytes) 000253 1767316660.836207 MIRROR_EXTEND to 3 mirrors 000255 1767316661.401308 trunc from 0x040000 to 0x00c4dc (0x33b23 bytes) 000256 1767316661.475592 mapread 0x002918 thru 0x00c4dc (0x9bc5 bytes) 000257 1767316661.491336 mapread 0x008685 thru 0x00c4dc (0x3e58 bytes) 000262 1767316661.495478 write 0x03c888 thru 0x03d443 (0xbbc bytes) 000263 1767316661.496805 trunc from 0x03d444 to 0x02cc23 (0x10820 bytes) 000265 1767316661.519162 write 0x03503f thru 0x03ffff (0xafc1 bytes) 000266 1767316661.525903 mapread 0x03cbf9 thru 0x03ffff (0x3407 bytes) 000267 1767316661.528389 write 0x006c2c thru 0x00ceef (0x62c4 bytes) 000271 1767316661.532891 read 0x025167 thru 0x0257ac (0x646 bytes) 000272 1767316661.544374 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x6c4c7b0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x6c4c7b0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000273 1767316661.823784 write 0x017801 thru 0x021197 (0x9997 bytes) 000274 1767316661.835778 write 0x004163 thru 0x00d5b6 (0x9454 bytes) 000275 1767316661.846145 mapwrite 0x01b00b thru 0x02193c (0x6932 bytes) 000282 1767316661.909135 mapwrite 0x02b011 thru 0x03a4cc (0xf4bc bytes) 000284 1767316661.977633 read 0x01d69c thru 0x029c40 (0xc5a5 bytes) 000290 1767316661.988081 mapwrite 0x013394 thru 0x018398 (0x5005 bytes) 000291 1767316662.033547 write 0x0134bc thru 0x017fc7 (0x4b0c bytes) 000293 1767316662.037900 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) 000295 1767316662.161925 read 0x0165f7 thru 0x0204b2 (0x9ebc bytes) 000296 1767316662.170591 mapwrite 0x039661 thru 0x03b52e (0x1ece bytes) 000297 1767316662.207081 mapwrite 0x01241f thru 0x01a81d (0x83ff bytes) 000298 1767316662.264880 write 0x0036ed thru 0x008c3d (0x5551 bytes) 000299 1767316662.274709 read 0x024b26 thru 0x02b5e9 (0x6ac4 bytes) 000300 1767316662.282875 trunc from 0x040000 to 0x00620a (0x39df5 bytes) 000301 1767316662.301380 mapwrite 0x020564 thru 0x026a64 (0x6501 bytes) 000302 1767316662.405719 read 0x00ae4d thru 0x0182be (0xd472 bytes) 000303 1767316662.420273 write 0x0090f6 thru 0x00e3a2 (0x52ad bytes) 000304 1767316662.421394 mapread 0x0211cc thru 0x024fe6 (0x3e1b bytes) 000305 1767316662.422173 trunc from 0x026a65 to 0x005d7d (0x20ce7 bytes) 000306 1767316662.435783 mapwrite 0x03295c thru 0x03ffff (0xd6a4 bytes) 000309 1767316662.537670 mapwrite 0x006e48 thru 0x014620 (0xd7d9 bytes) 000310 1767316662.677061 trunc from 0x040000 to 0x030a8e (0xf571 bytes) 000311 1767316662.705142 trunc from 0x030a8f to 0x0005fd (0x30491 bytes) 000312 1767316662.726086 mapwrite 0x00927d thru 0x016110 (0xce94 bytes) 000313 1767316662.847571 mapread 0x00f85e thru 0x016110 (0x68b3 bytes) 000316 1767316662.849445 write 0x02fbdc thru 0x03f0df (0xf504 bytes) 000319 1767316662.851536 read 0x0368b2 thru 0x03f0df (0x882e bytes) 000321 1767316662.852596 mapwrite 0x001917 thru 0x00990d (0x7ff7 bytes) 000323 1767316662.924301 trunc from 0x03f0e0 to 0x024bb8 (0x1a527 bytes) 000325 1767316662.947164 mapread 0x016f99 thru 0x02025c (0x92c4 bytes) 000328 1767316662.951037 mapwrite 0x037a6e thru 0x03ffff (0x8592 bytes) 000330 1767316663.066039 trunc from 0x040000 to 0x014acc (0x2b533 bytes) 000331 1767316663.081367 read 0x00576b thru 0x014acc (0xf362 bytes) 000334 1767316663.082758 mapread 0x014159 thru 0x014acc (0x974 bytes) 000335 1767316663.083722 trunc from 0x014acd to 0x008aaa (0xc022 bytes) 000340 1767316663.112775 write 0x019e83 thru 0x0293ef (0xf56d bytes) 000343 1767316663.116972 trunc from 0x0293f0 to 0x02f43a (0x604b bytes) 000345 1767316663.141673 trunc from 0x02f43b to 0x032b8c (0x3752 bytes) 000348 1767316663.165503 MIRROR_EXTEND to 2 mirrors 000349 1767316663.712742 write 0x022a9c thru 0x02b1b8 (0x871d bytes) 000350 1767316663.808789 read 0x027c3a thru 0x02d732 (0x5af9 bytes) 000351 1767316663.813831 trunc from 0x032b8d to 0x016c11 (0x1bf7b bytes) 000353 1767316663.842446 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x16c12): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16c12): Mirror 2: 0x5f3f75e0 (stale) CRC-32 checksum value for chunk [0, 0x16c12): Mirror 1: 0x5f3f75e0 Mirror 3: 0x5f3f75e0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16c12: skipped 000354 1767316664.598949 mapread 0x016b65 thru 0x016c11 (0xad bytes) 000355 1767316664.647620 trunc from 0x016c12 to 0x033827 (0x1cc16 bytes) 000358 1767316664.746971 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33828): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33828): Mirror 2: 0xda39aa1e (stale) Mirror 3: 0x63f1cd3c (stale) CRC-32 checksum value for chunk [0, 0x33828): Mirror 1: 0x63f1cd3c Mirror 4: 0x63f1cd3c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33828: skipped 000359 1767316665.593064 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x33828): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33828): Mirror 2: 0xda39aa1e (stale) Mirror 3: 0x63f1cd3c (stale) CRC-32 checksum value for chunk [0, 0x33828): Mirror 1: 0x63f1cd3c Mirror 4: 0x63f1cd3c Mirror 5: 0x63f1cd3c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33828: skipped 000361 1767316666.441361 mapwrite 0x018c1d thru 0x01ab48 (0x1f2c bytes) 000362 1767316666.630697 write 0x00b136 thru 0x015d24 (0xabef bytes) 000364 1767316666.640744 write 0x008ca2 thru 0x013341 (0xa6a0 bytes) 000367 1767316666.648055 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x33828): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33828): Mirror 2: 0xda39aa1e (stale) Mirror 3: 0x63f1cd3c (stale) Mirror 4: 0x63f1cd3c (stale) Mirror 5: 0x63f1cd3c (stale) CRC-32 checksum value for chunk [0, 0x33828): Mirror 1: 0xd6fe5bb Mirror 6: 0xd6fe5bb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33828: skipped 000370 1767316667.617826 trunc from 0x033828 to 0x001afc (0x31d2b bytes) truncating to largest ever: 0x3da46 000371 1767316667.772905 trunc from 0x001afd to 0x03da45 (0x3bf49 bytes) 000373 1767316667.826294 mapwrite 0x019a63 thru 0x020bb2 (0x7150 bytes) 000374 1767316667.977576 read 0x007073 thru 0x00c0da (0x5068 bytes) 000375 1767316667.987402 mapread 0x008add thru 0x00d73e (0x4c62 bytes) 000376 1767316668.016366 MIRROR_RESYNC 6 mirrors 000377 1767316669.641976 trunc from 0x03da46 to 0x02ba71 (0x11fd4 bytes) 000380 1767316669.735448 trunc from 0x02ba72 to 0x0273dc (0x4695 bytes) 000384 1767316669.766473 write 0x02ec26 thru 0x032fc4 (0x439f bytes) 000385 1767316669.768068 write 0x006d5a thru 0x00bc26 (0x4ecd bytes) 000386 1767316669.782073 mapread 0x0166cb thru 0x020b5f (0xa495 bytes) 000388 1767316669.794939 read 0x0269c4 thru 0x02bd20 (0x535d bytes) 000392 1767316669.799955 write 0x0384d3 thru 0x03ffff (0x7b2d bytes) 000396 1767316669.801701 write 0x030b3e thru 0x037e1d (0x72e0 bytes) 000399 1767316669.808491 mapwrite 0x014dbf thru 0x01598b (0xbcd bytes) 000400 1767316669.893811 trunc from 0x040000 to 0x026230 (0x19dcf bytes) 000403 1767316669.911480 trunc from 0x026231 to 0x02dfa3 (0x7d73 bytes) 000404 1767316669.930122 mapwrite 0x008438 thru 0x00e9ae (0x6577 bytes) 000406 1767316669.980642 trunc from 0x02dfa4 to 0x003445 (0x2ab5e bytes) 000407 1767316670.004410 mapwrite 0x02ee9a thru 0x03d5dc (0xe743 bytes) 000408 1767316670.243210 mapwrite 0x02bd0f thru 0x03b8cc (0xfbbe bytes) 000410 1767316670.333628 mapread 0x020849 thru 0x02b3ed (0xaba5 bytes) 000412 1767316670.359220 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3d5dd): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d5dd): Mirror 2: 0x23822f10 (stale) Mirror 3: 0x23822f10 (stale) Mirror 4: 0x23822f10 (stale) Mirror 5: 0x23822f10 (stale) Mirror 6: 0x23822f10 (stale) CRC-32 checksum value for chunk [0, 0x3d5dd): Mirror 1: 0xee50b5ad Mirror 7: 0xee50b5ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d5dd: skipped 000414 1767316671.641893 mapwrite 0x039bbf thru 0x03e581 (0x49c3 bytes) 000415 1767316671.904756 mapread 0x01d2ab thru 0x02ca71 (0xf7c7 bytes) 000418 1767316671.959323 mapwrite 0x02ffa2 thru 0x0358cd (0x592c bytes) 000419 1767316672.161966 read 0x029c1d thru 0x02c77d (0x2b61 bytes) 000421 1767316672.163954 trunc from 0x03e582 to 0x01c9c0 (0x21bc1 bytes) 000422 1767316672.219025 mapwrite 0x02e7ec thru 0x03df77 (0xf78c bytes) 000423 1767316672.361477 mapwrite 0x03ef7e thru 0x03ffff (0x1082 bytes) 000424 1767316672.444945 trunc from 0x040000 to 0x025995 (0x1a66a bytes) 000427 1767316672.477694 read 0x01dcd5 thru 0x025995 (0x7cc1 bytes) 000434 1767316672.507002 mapread 0x022c3b thru 0x025995 (0x2d5b bytes) 000435 1767316672.509505 read 0x0048a6 thru 0x0079ed (0x3148 bytes) 000436 1767316672.511185 mapread 0x021de7 thru 0x025995 (0x3baf bytes) 000437 1767316672.512806 read 0x01e0d6 thru 0x01f233 (0x115e bytes) 000438 1767316672.513447 mapwrite 0x00cad6 thru 0x013f97 (0x74c2 bytes) 000439 1767316672.638941 read 0x01746d thru 0x022c52 (0xb7e6 bytes) 000442 1767316672.645401 mapwrite 0x00c96b thru 0x01c33f (0xf9d5 bytes) 000445 1767316672.797665 write 0x029cd5 thru 0x031353 (0x767f bytes) 000446 1767316672.798757 read 0x01b584 thru 0x025c5e (0xa6db bytes) 000448 1767316672.800200 write 0x019328 thru 0x028390 (0xf069 bytes) 000449 1767316672.804792 write 0x021b63 thru 0x024283 (0x2721 bytes) 000450 1767316672.806388 mapwrite 0x00e80c thru 0x01531a (0x6b0f bytes) 000451 1767316672.850146 mapwrite 0x032658 thru 0x03c7b2 (0xa15b bytes) 000453 1767316672.915869 mapwrite 0x037d41 thru 0x03ffff (0x82bf bytes) 000455 1767316672.961465 trunc from 0x040000 to 0x03a83f (0x57c0 bytes) 000456 1767316672.978907 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a840): Mirror 3: 0xe9c6bd8c (stale) Mirror 4: 0xe9c6bd8c (stale) Mirror 5: 0xe9c6bd8c (stale) Mirror 6: 0xe9c6bd8c (stale) Mirror 7: 0x73744cd8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a840): Mirror 3: 0xe9c6bd8c (stale) Mirror 4: 0xe9c6bd8c (stale) Mirror 5: 0xe9c6bd8c (stale) Mirror 6: 0xe9c6bd8c (stale) Mirror 7: 0x73744cd8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a840: skipped 000458 1767316673.346200 mapread 0x018ad9 thru 0x021c02 (0x912a bytes) 000459 1767316673.358981 trunc from 0x03a840 to 0x018fe7 (0x21858 bytes) 000461 1767316673.375525 mapwrite 0x01c48b thru 0x02580b (0x9381 bytes) 000462 1767316673.528772 read 0x022f42 thru 0x02580b (0x28ca bytes) 000463 1767316673.531104 mapread 0x01b62a thru 0x0237a4 (0x817b bytes) 000464 1767316673.536774 mapwrite 0x036798 thru 0x03cfb4 (0x681d bytes) 000465 1767316673.669819 mapread 0x01a771 thru 0x026ae8 (0xc378 bytes) 000466 1767316673.694585 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3cfb5): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cfb5): Mirror 3: 0x70d6f8dd (stale) Mirror 4: 0x70d6f8dd (stale) Mirror 5: 0x70d6f8dd (stale) Mirror 6: 0x70d6f8dd (stale) Mirror 7: 0xad37ffb2 (stale) CRC-32 checksum value for chunk [0, 0x3cfb5): Mirror 1: 0x6fbb6fb6 Mirror 8: 0x6fbb6fb6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cfb5: skipped 000470 1767316674.578558 mapread 0x002ce9 thru 0x00660b (0x3923 bytes) 000472 1767316674.612416 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3cfb5): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cfb5): Mirror 3: 0x70d6f8dd (stale) Mirror 4: 0x70d6f8dd (stale) Mirror 5: 0x70d6f8dd (stale) Mirror 6: 0x70d6f8dd (stale) Mirror 7: 0xad37ffb2 (stale) CRC-32 checksum value for chunk [0, 0x3cfb5): Mirror 1: 0x6fbb6fb6 Mirror 8: 0x6fbb6fb6 Mirror 9: 0x6fbb6fb6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cfb5: skipped 000474 1767316675.554586 trunc from 0x03cfb5 to 0x005af5 (0x374bf bytes) 000475 1767316675.647094 MIRROR_RESYNC 8 mirrors 000476 1767316677.588058 mapread 0x001091 thru 0x005af5 (0x4a65 bytes) 000477 1767316677.613460 write 0x02e110 thru 0x03b9d9 (0xd8ca bytes) 000478 1767316677.698532 write 0x038087 thru 0x03e5a7 (0x6521 bytes) 000479 1767316677.700730 trunc from 0x03e5a8 to 0x034e25 (0x9782 bytes) 000480 1767316677.741497 read 0x02f361 thru 0x0321af (0x2e4f bytes) 000483 1767316677.742694 mapread 0x001d57 thru 0x00653c (0x47e6 bytes) 000484 1767316677.749969 read 0x0313ec thru 0x034e25 (0x3a3a bytes) 000485 1767316677.750498 mapwrite 0x03bd12 thru 0x03ffff (0x42ee bytes) 000488 1767316677.806398 read 0x001e39 thru 0x008d3a (0x6f02 bytes) 000492 1767316677.807998 mapread 0x00aa18 thru 0x012382 (0x796b bytes) 000493 1767316677.808946 mapwrite 0x01bd62 thru 0x01f412 (0x36b1 bytes) 000497 1767316677.837534 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xb282940a (stale) Mirror 4: 0xb282940a (stale) Mirror 5: 0xb282940a (stale) Mirror 6: 0xb282940a (stale) Mirror 7: 0xb282940a (stale) Mirror 8: 0xb282940a (stale) Mirror 9: 0xb282940a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcb4adb50 Mirror 10: 0xcb4adb50 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000499 1767316678.739129 MIRROR_RESYNC 9 mirrors 000500 1767316679.831167 trunc from 0x040000 to 0x021b9a (0x1e465 bytes) 000501 1767316679.918227 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x21b9b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21b9b): Mirror 3: 0xdbd0d3c2 (stale) Mirror 4: 0xdbd0d3c2 (stale) Mirror 5: 0xdbd0d3c2 (stale) Mirror 6: 0xdbd0d3c2 (stale) Mirror 7: 0xdbd0d3c2 (stale) Mirror 8: 0xdbd0d3c2 (stale) Mirror 9: 0xdbd0d3c2 (stale) Mirror 10: 0xdbd0d3c2 (stale) CRC-32 checksum value for chunk [0, 0x21b9b): Mirror 1: 0xdbd0d3c2 Mirror 11: 0xdbd0d3c2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21b9b: skipped 000503 1767316680.727528 trunc from 0x021b9b to 0x02a92a (0x8d90 bytes) 000505 1767316680.792932 trunc from 0x02a92b to 0x00c1d4 (0x1e756 bytes) 000506 1767316680.813019 write 0x03436b thru 0x03966a (0x5300 bytes) 000508 1767316680.821776 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3966b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3966b): Mirror 3: 0xb62a0950 (stale) Mirror 4: 0xb62a0950 (stale) Mirror 5: 0xb62a0950 (stale) Mirror 6: 0xb62a0950 (stale) Mirror 7: 0xb62a0950 (stale) Mirror 8: 0xb62a0950 (stale) Mirror 9: 0xb62a0950 (stale) Mirror 10: 0xb62a0950 (stale) Mirror 11: 0xc0b661 (stale) CRC-32 checksum value for chunk [0, 0x3966b): Mirror 1: 0xeff8f1bb Mirror 12: 0xeff8f1bb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3966b: skipped 000511 1767316681.667527 mapread 0x02a908 thru 0x031c20 (0x7319 bytes) 000512 1767316681.694112 trunc from 0x03966b to 0x03651d (0x314d bytes) 000513 1767316681.776904 write 0x014ce1 thru 0x01a897 (0x5bb7 bytes) 000514 1767316681.785393 read 0x0344db thru 0x03651d (0x2043 bytes) 000516 1767316681.791428 mapwrite 0x03f150 thru 0x03ffff (0xeb0 bytes) 000517 1767316681.854305 trunc from 0x040000 to 0x034865 (0xb79a bytes) 000518 1767316681.873541 MIRROR_RESYNC 11 mirrors 000520 1767316683.235792 read 0x005b29 thru 0x013ced (0xe1c5 bytes) 000521 1767316683.248238 MIRROR_RESYNC 11 mirrors 000523 1767316683.921545 write 0x03fcbe thru 0x03ffff (0x342 bytes) 000524 1767316683.984703 trunc from 0x040000 to 0x03351a (0xcae5 bytes) 000525 1767316684.012444 trunc from 0x03351b to 0x03cd34 (0x981a bytes) 000526 1767316684.025753 mapread 0x00f770 thru 0x010862 (0x10f3 bytes) 000527 1767316684.036640 mapwrite 0x03b223 thru 0x03ffff (0x4ddd bytes) 000528 1767316684.090201 MIRROR_RESYNC 11 mirrors 000530 1767316685.534290 write 0x034a19 thru 0x03956c (0x4b54 bytes) 000531 1767316685.588526 read 0x01d341 thru 0x02a658 (0xd318 bytes) 000532 1767316685.598634 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7cb97e92 (stale) Mirror 4: 0x7cb97e92 (stale) Mirror 5: 0x7cb97e92 (stale) Mirror 6: 0x7cb97e92 (stale) Mirror 7: 0x7cb97e92 (stale) Mirror 8: 0x7cb97e92 (stale) Mirror 9: 0x7cb97e92 (stale) Mirror 10: 0x7cb97e92 (stale) Mirror 11: 0x7cb97e92 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7cb97e92 (stale) Mirror 4: 0x7cb97e92 (stale) Mirror 5: 0x7cb97e92 (stale) Mirror 6: 0x7cb97e92 (stale) Mirror 7: 0x7cb97e92 (stale) Mirror 8: 0x7cb97e92 (stale) Mirror 9: 0x7cb97e92 (stale) Mirror 10: 0x7cb97e92 (stale) Mirror 11: 0x7cb97e92 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000533 1767316686.213683 read 0x02ae49 thru 0x02b3af (0x567 bytes) 000542 1767316686.231553 write 0x006778 thru 0x00c6f1 (0x5f7a bytes) 000545 1767316686.272674 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7cb97e92 (stale) Mirror 4: 0x7cb97e92 (stale) Mirror 5: 0x7cb97e92 (stale) Mirror 6: 0x7cb97e92 (stale) Mirror 7: 0x7cb97e92 (stale) Mirror 8: 0x7cb97e92 (stale) Mirror 9: 0x7cb97e92 (stale) Mirror 10: 0x7cb97e92 (stale) Mirror 11: 0x7cb97e92 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdbdf2486 Mirror 12: 0xdbdf2486 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000549 1767316687.751279 trunc from 0x040000 to 0x011918 (0x2e6e7 bytes) 000550 1767316687.959040 write 0x00f067 thru 0x014dbe (0x5d58 bytes) 000551 1767316687.969244 read 0x000a6e thru 0x00896f (0x7f02 bytes) 000552 1767316687.987572 mapread 0x007726 thru 0x009db4 (0x268f bytes) 000554 1767316687.994565 trunc from 0x014dbf to 0x003702 (0x116bc bytes) 000555 1767316688.040454 read 0x000947 thru 0x00362a (0x2ce4 bytes) 000556 1767316688.041589 trunc from 0x003703 to 0x0332df (0x2fbdd bytes) 000557 1767316688.106747 trunc from 0x0332e0 to 0x001df8 (0x314e7 bytes) 000558 1767316688.148823 mapwrite 0x01a0b0 thru 0x0242dd (0xa22e bytes) 000560 1767316688.381424 read 0x00fe7c thru 0x014f42 (0x50c7 bytes) 000561 1767316688.385988 MIRROR_RESYNC 11 mirrors 000566 1767316690.646410 mapread 0x0040c3 thru 0x00ac27 (0x6b65 bytes) 000568 1767316690.671580 mapread 0x0182fc thru 0x0242dd (0xbfe2 bytes) 000569 1767316690.685752 trunc from 0x0242de to 0x022f62 (0x137b bytes) 000571 1767316690.813790 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22f63): Mirror 3: 0x48d98123 (stale) Mirror 4: 0x48d98123 (stale) Mirror 5: 0x48d98123 (stale) Mirror 7: 0x48d98123 (stale) Mirror 8: 0x48d98123 (stale) Mirror 9: 0x48d98123 (stale) Mirror 10: 0x48d98123 (stale) Mirror 11: 0x48d98123 (stale) Mirror 12: 0x48d98123 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22f63): Mirror 3: 0x48d98123 (stale) Mirror 4: 0x48d98123 (stale) Mirror 5: 0x48d98123 (stale) Mirror 7: 0x48d98123 (stale) Mirror 8: 0x48d98123 (stale) Mirror 9: 0x48d98123 (stale) Mirror 10: 0x48d98123 (stale) Mirror 11: 0x48d98123 (stale) Mirror 12: 0x48d98123 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22f63: skipped 000572 1767316691.497313 mapread 0x021f84 thru 0x022f62 (0xfdf bytes) 000574 1767316691.514982 write 0x03524d thru 0x03932f (0x40e3 bytes) 000575 1767316691.535058 MIRROR_RESYNC 10 mirrors 000577 1767316693.538027 trunc from 0x039330 to 0x03482a (0x4b05 bytes) 000581 1767316693.661552 read 0x00f9b1 thru 0x01044f (0xa9f bytes) 000582 1767316693.673557 MIRROR_RESYNC 10 mirrors 000584 1767316695.154471 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) 000585 1767316695.545511 read 0x006907 thru 0x00b139 (0x4833 bytes) 000586 1767316695.553336 mapwrite 0x032873 thru 0x03ffff (0xd78d bytes) 000588 1767316695.694613 write 0x0284d1 thru 0x033ccc (0xb7fc bytes) 000592 1767316695.702598 write 0x01de34 thru 0x022ef1 (0x50be bytes) 000593 1767316695.710039 mapwrite 0x01aab9 thru 0x027e3a (0xd382 bytes) 000595 1767316695.761461 write 0x03992f thru 0x03ffff (0x66d1 bytes) 000596 1767316695.762297 read 0x024141 thru 0x033954 (0xf814 bytes) 000597 1767316695.762737 mapread 0x0371db thru 0x03d782 (0x65a8 bytes) 000598 1767316695.766669 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xd5678181 (stale) Mirror 5: 0xd5678181 (stale) Mirror 7: 0xd5678181 (stale) Mirror 8: 0xd5678181 (stale) Mirror 9: 0xd5678181 (stale) Mirror 10: 0xd5678181 (stale) Mirror 11: 0xd5678181 (stale) Mirror 12: 0xd5678181 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9a7cc4a9 Mirror 13: 0x9a7cc4a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000599 1767316697.040544 read 0x027a91 thru 0x031550 (0x9ac0 bytes) 000600 1767316697.062026 trunc from 0x040000 to 0x0271e6 (0x18e19 bytes) 000601 1767316697.168301 MIRROR_RESYNC 10 mirrors 000602 1767316698.948436 read 0x004d6a thru 0x00aa62 (0x5cf9 bytes) 000604 1767316698.959527 mapwrite 0x02e7f5 thru 0x031e96 (0x36a2 bytes) 000606 1767316699.087472 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31e97): Mirror 4: 0xf6f56990 (stale) Mirror 5: 0xf6f56990 (stale) Mirror 8: 0xf6f56990 (stale) Mirror 9: 0xf6f56990 (stale) Mirror 10: 0xf6f56990 (stale) Mirror 11: 0xf6f56990 (stale) Mirror 12: 0xf6f56990 (stale) Mirror 13: 0xf6f56990 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31e97): Mirror 4: 0xf6f56990 (stale) Mirror 5: 0xf6f56990 (stale) Mirror 8: 0xf6f56990 (stale) Mirror 9: 0xf6f56990 (stale) Mirror 10: 0xf6f56990 (stale) Mirror 11: 0xf6f56990 (stale) Mirror 12: 0xf6f56990 (stale) Mirror 13: 0xf6f56990 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31e97: skipped 000607 1767316699.634121 trunc from 0x031e97 to 0x02204d (0xfe49 bytes) 000608 1767316699.656693 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2204e): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2204e): Mirror 4: 0x1ebd554b (stale) Mirror 5: 0x1ebd554b (stale) Mirror 8: 0x1ebd554b (stale) Mirror 9: 0x1ebd554b (stale) Mirror 10: 0x1ebd554b (stale) Mirror 11: 0x1ebd554b (stale) Mirror 12: 0x1ebd554b (stale) Mirror 13: 0x1ebd554b (stale) CRC-32 checksum value for chunk [0, 0x2204e): Mirror 1: 0x1ebd554b Mirror 14: 0x1ebd554b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2204e: skipped 000610 1767316700.919059 mapread 0x010c86 thru 0x01837f (0x76fa bytes) 000611 1767316700.974020 trunc from 0x02204e to 0x0323a8 (0x1035b bytes) 000612 1767316701.116146 write 0x01be35 thru 0x0250b4 (0x9280 bytes) 000615 1767316701.152668 write 0x022885 thru 0x027d2c (0x54a8 bytes) 000617 1767316701.192097 trunc from 0x0323a9 to 0x02e202 (0x41a6 bytes) 000618 1767316701.225291 mapwrite 0x0199db thru 0x024d19 (0xb33f bytes) 000621 1767316701.362764 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2e203): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e203): Mirror 4: 0x724499ba (stale) Mirror 5: 0x724499ba (stale) Mirror 8: 0x724499ba (stale) Mirror 9: 0x724499ba (stale) Mirror 10: 0x724499ba (stale) Mirror 11: 0x724499ba (stale) Mirror 12: 0x724499ba (stale) Mirror 13: 0x724499ba (stale) Mirror 14: 0x643d4a53 (stale) CRC-32 checksum value for chunk [0, 0x2e203): Mirror 1: 0x2bc8da56 Mirror 15: 0x2bc8da56 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e203: skipped 000622 1767316702.206054 mapread 0x017e77 thru 0x0231e7 (0xb371 bytes) 000624 1767316702.265258 mapwrite 0x02d107 thru 0x039520 (0xc41a bytes) 000627 1767316702.494792 mapread 0x00ccc8 thru 0x00fde9 (0x3122 bytes) 000628 1767316702.505141 read 0x026f3f thru 0x026fe1 (0xa3 bytes) 000629 1767316702.509504 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39521): Mirror 4: 0x3381ed74 (stale) Mirror 5: 0x3381ed74 (stale) Mirror 9: 0x3381ed74 (stale) Mirror 10: 0x3381ed74 (stale) Mirror 11: 0x3381ed74 (stale) Mirror 12: 0x3381ed74 (stale) Mirror 13: 0x3381ed74 (stale) Mirror 14: 0x49307189 (stale) Mirror 15: 0x66477d05 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39521): Mirror 4: 0x3381ed74 (stale) Mirror 5: 0x3381ed74 (stale) Mirror 9: 0x3381ed74 (stale) Mirror 10: 0x3381ed74 (stale) Mirror 11: 0x3381ed74 (stale) Mirror 12: 0x3381ed74 (stale) Mirror 13: 0x3381ed74 (stale) Mirror 14: 0x49307189 (stale) Mirror 15: 0x66477d05 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39521: skipped 000630 1767316703.205347 trunc from 0x039521 to 0x025334 (0x141ec bytes) 000631 1767316703.222322 mapread 0x012041 thru 0x015b07 (0x3ac7 bytes) 000632 1767316703.239752 trunc from 0x025335 to 0x032557 (0xd223 bytes) 000633 1767316703.253335 write 0x03edd0 thru 0x03ffff (0x1230 bytes) 000634 1767316703.254594 mapread 0x0001cc thru 0x004e73 (0x4ca8 bytes) 000635 1767316703.264447 read 0x03c221 thru 0x03ffff (0x3ddf bytes) 000636 1767316703.274452 mapread 0x0162b4 thru 0x01f7a1 (0x94ee bytes) 000638 1767316703.284358 read 0x03f71a thru 0x03ffff (0x8e6 bytes) 000639 1767316703.290234 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x3afed90f (stale) Mirror 5: 0x3afed90f (stale) Mirror 9: 0x3afed90f (stale) Mirror 10: 0x3afed90f (stale) Mirror 11: 0x3afed90f (stale) Mirror 12: 0x3afed90f (stale) Mirror 13: 0x3afed90f (stale) Mirror 14: 0x57c3928d (stale) Mirror 15: 0x30386354 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9a2aa14b Mirror 16: 0x9a2aa14b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000640 1767316704.104895 trunc from 0x040000 to 0x030e58 (0xf1a7 bytes) 000641 1767316704.229258 write 0x03a994 thru 0x03c1cc (0x1839 bytes) 000645 1767316704.234010 mapread 0x004382 thru 0x01263b (0xe2ba bytes) 000646 1767316704.261673 read 0x01baa2 thru 0x01fccd (0x422c bytes) 000647 1767316704.270409 read 0x0054fb thru 0x0113cb (0xbed1 bytes) 000648 1767316704.271217 read 0x00268c thru 0x009f4d (0x78c2 bytes) 000651 1767316704.273415 trunc from 0x03c1cd to 0x006933 (0x35899 bytes) 000654 1767316704.288775 mapread 0x001326 thru 0x001ccf (0x9aa bytes) 000657 1767316704.291542 mapread 0x0024d6 thru 0x006933 (0x445e bytes) 000658 1767316704.293224 write 0x037372 thru 0x03b90f (0x459e bytes) 000659 1767316704.295348 mapwrite 0x01ea99 thru 0x02d9d8 (0xef40 bytes) 000660 1767316704.354274 read 0x030e04 thru 0x030e5e (0x5b bytes) 000665 1767316704.356621 mapwrite 0x026e25 thru 0x02f3f9 (0x85d5 bytes) 000667 1767316704.402116 read 0x0259de thru 0x02ff96 (0xa5b9 bytes) 000669 1767316704.404012 mapread 0x009da9 thru 0x014e51 (0xb0a9 bytes) 000670 1767316704.405421 mapread 0x023cbe thru 0x0303f1 (0xc734 bytes) 000671 1767316704.407253 mapwrite 0x025431 thru 0x02e41d (0x8fed bytes) 000674 1767316704.456058 mapread 0x0301c9 thru 0x031c46 (0x1a7e bytes) 000676 1767316704.463217 MIRROR_RESYNC 11 mirrors 000679 1767316706.013170 MIRROR_RESYNC 11 mirrors 000682 1767316706.784837 MIRROR_EXTEND to 12 mirrors 000683 1767316707.660705 write 0x01e85f thru 0x02b6ae (0xce50 bytes) 000684 1767316707.751371 write 0x02eae0 thru 0x0397d4 (0xacf5 bytes) 000686 1767316707.771104 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3b910): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b910): Mirror 4: 0x7063107b (stale) Mirror 5: 0x7063107b (stale) Mirror 9: 0x7063107b (stale) Mirror 10: 0x7063107b (stale) Mirror 11: 0x7063107b (stale) Mirror 12: 0x7063107b (stale) Mirror 13: 0x7063107b (stale) Mirror 14: 0x7063107b (stale) Mirror 15: 0x7063107b (stale) Mirror 16: 0x7063107b (stale) Mirror 17: 0x7063107b (stale) CRC-32 checksum value for chunk [0, 0x3b910): Mirror 1: 0x21096cb6 Mirror 18: 0x21096cb6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b910: skipped 000687 1767316709.014095 read 0x023c75 thru 0x028e83 (0x520f bytes) 000691 1767316709.085344 mapread 0x00be65 thru 0x00c05d (0x1f9 bytes) 000692 1767316709.113105 read 0x019ec5 thru 0x0264cb (0xc607 bytes) 000694 1767316709.170006 trunc from 0x03b910 to 0x03a3f3 (0x151c bytes) 000695 1767316709.362139 trunc from 0x03a3f4 to 0x026b42 (0x138b1 bytes) 000696 1767316709.418920 read 0x00b64e thru 0x01483b (0x91ee bytes) 000698 1767316709.440468 write 0x00d502 thru 0x01c2ed (0xedec bytes) 000699 1767316709.461942 MIRROR_SPLIT mirror 8 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26b43): Mirror 4: 0xa594db3d (stale) Mirror 5: 0xa594db3d (stale) Mirror 9: 0xa594db3d (stale) Mirror 10: 0xa594db3d (stale) Mirror 11: 0xa594db3d (stale) Mirror 12: 0xa594db3d (stale) Mirror 13: 0xa594db3d (stale) Mirror 15: 0xa594db3d (stale) Mirror 16: 0xa594db3d (stale) Mirror 17: 0xa594db3d (stale) Mirror 18: 0x6b07ab39 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26b43): Mirror 4: 0xa594db3d (stale) Mirror 5: 0xa594db3d (stale) Mirror 9: 0xa594db3d (stale) Mirror 10: 0xa594db3d (stale) Mirror 11: 0xa594db3d (stale) Mirror 12: 0xa594db3d (stale) Mirror 13: 0xa594db3d (stale) Mirror 15: 0xa594db3d (stale) Mirror 16: 0xa594db3d (stale) Mirror 17: 0xa594db3d (stale) Mirror 18: 0x6b07ab39 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26b43: skipped 000700 1767316710.263082 mapwrite 0x034418 thru 0x034e33 (0xa1c bytes) 000701 1767316710.340231 mapread 0x00a419 thru 0x00d58b (0x3173 bytes) 000702 1767316710.355307 trunc from 0x034e34 to 0x01fc1e (0x15215 bytes) 000706 1767316710.376538 mapwrite 0x02f8c3 thru 0x03b4e8 (0xbc26 bytes) 000707 1767316710.439977 mapwrite 0x00fb45 thru 0x01c625 (0xcae1 bytes) 000708 1767316710.495737 write 0x01b741 thru 0x02aea7 (0xf767 bytes) 000709 1767316710.498009 trunc from 0x03b4e9 to 0x0391a9 (0x233f bytes) 000710 1767316710.521168 mapwrite 0x03c340 thru 0x03ffff (0x3cc0 bytes) 000711 1767316710.590222 mapwrite 0x00dd38 thru 0x0193b1 (0xb67a bytes) 000713 1767316710.637811 write 0x00924a thru 0x018ed1 (0xfc88 bytes) 000715 1767316710.642183 read 0x027147 thru 0x035114 (0xdfce bytes) 000716 1767316710.649153 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xee7667a8 (stale) Mirror 5: 0xee7667a8 (stale) Mirror 9: 0xee7667a8 (stale) Mirror 10: 0xee7667a8 (stale) Mirror 11: 0xee7667a8 (stale) Mirror 12: 0xee7667a8 (stale) Mirror 13: 0xee7667a8 (stale) Mirror 15: 0xee7667a8 (stale) Mirror 16: 0xee7667a8 (stale) Mirror 18: 0x1aa3710 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xee7667a8 (stale) Mirror 5: 0xee7667a8 (stale) Mirror 9: 0xee7667a8 (stale) Mirror 10: 0xee7667a8 (stale) Mirror 11: 0xee7667a8 (stale) Mirror 12: 0xee7667a8 (stale) Mirror 13: 0xee7667a8 (stale) Mirror 15: 0xee7667a8 (stale) Mirror 16: 0xee7667a8 (stale) Mirror 18: 0x1aa3710 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000717 1767316711.225732 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xee7667a8 (stale) Mirror 5: 0xee7667a8 (stale) Mirror 9: 0xee7667a8 (stale) Mirror 10: 0xee7667a8 (stale) Mirror 11: 0xee7667a8 (stale) Mirror 12: 0xee7667a8 (stale) Mirror 13: 0xee7667a8 (stale) Mirror 15: 0xee7667a8 (stale) Mirror 16: 0xee7667a8 (stale) Mirror 18: 0x1aa3710 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfffc3f6c Mirror 19: 0xfffc3f6c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000719 1767316712.542156 mapwrite 0x038a6e thru 0x03ffff (0x7592 bytes) 000720 1767316712.801420 write 0x036a89 thru 0x03b55f (0x4ad7 bytes) 000721 1767316712.803034 mapwrite 0x007574 thru 0x00d274 (0x5d01 bytes) 000722 1767316712.894654 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xee7667a8 (stale) Mirror 5: 0xee7667a8 (stale) Mirror 9: 0xee7667a8 (stale) Mirror 10: 0xee7667a8 (stale) Mirror 11: 0xee7667a8 (stale) Mirror 12: 0xee7667a8 (stale) Mirror 13: 0xee7667a8 (stale) Mirror 15: 0xee7667a8 (stale) Mirror 16: 0xee7667a8 (stale) Mirror 18: 0x1aa3710 (stale) Mirror 19: 0xfffc3f6c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa80249cc Mirror 20: 0xa80249cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000723 1767316714.049741 read 0x0112bc thru 0x017f20 (0x6c65 bytes) 000724 1767316714.068033 trunc from 0x040000 to 0x02eb76 (0x11489 bytes) 000726 1767316714.177549 read 0x00ae0e thru 0x015e8b (0xb07e bytes) 000727 1767316714.192248 write 0x02211c thru 0x023e20 (0x1d05 bytes) 000728 1767316714.193448 trunc from 0x02eb77 to 0x0046b3 (0x2a4c3 bytes) 000729 1767316714.223975 mapwrite 0x015edf thru 0x024b27 (0xec49 bytes) 000730 1767316714.439291 MIRROR_RESYNC 13 mirrors 000733 1767316716.116028 mapread 0x00e652 thru 0x0167b3 (0x8162 bytes) 000734 1767316716.136009 mapread 0x002a2e thru 0x00813d (0x5710 bytes) 000736 1767316716.150687 MIRROR_EXTEND to 14 mirrors 000737 1767316716.940539 MIRROR_RESYNC 14 mirrors 000740 1767316717.468520 write 0x018c07 thru 0x0242dd (0xb6d7 bytes) 000741 1767316717.542520 read 0x01d0f8 thru 0x024b27 (0x7a30 bytes) 000742 1767316717.543385 write 0x0082d4 thru 0x009e96 (0x1bc3 bytes) 000743 1767316717.553672 mapwrite 0x018721 thru 0x0192f2 (0xbd2 bytes) 000745 1767316717.618540 mapread 0x01708a thru 0x020a56 (0x99cd bytes) 000746 1767316717.628548 trunc from 0x024b28 to 0x0098a7 (0x1b280 bytes) 000748 1767316717.660493 mapread 0x0007c8 thru 0x00649d (0x5cd6 bytes) 000750 1767316717.668613 read 0x002915 thru 0x0098a7 (0x6f93 bytes) 000753 1767316717.670961 trunc from 0x0098a8 to 0x029083 (0x1f7dc bytes) 000754 1767316717.696747 MIRROR_SPLIT mirror 9 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29084): Mirror 4: 0x4698394e (stale) Mirror 5: 0x4698394e (stale) Mirror 9: 0x4698394e (stale) Mirror 10: 0x4698394e (stale) Mirror 11: 0x4698394e (stale) Mirror 12: 0x4698394e (stale) Mirror 13: 0x4698394e (stale) Mirror 15: 0x4698394e (stale) Mirror 18: 0x4698394e (stale) Mirror 19: 0x4698394e (stale) Mirror 20: 0x4698394e (stale) Mirror 21: 0x4698394e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29084): Mirror 4: 0x4698394e (stale) Mirror 5: 0x4698394e (stale) Mirror 9: 0x4698394e (stale) Mirror 10: 0x4698394e (stale) Mirror 11: 0x4698394e (stale) Mirror 12: 0x4698394e (stale) Mirror 13: 0x4698394e (stale) Mirror 15: 0x4698394e (stale) Mirror 18: 0x4698394e (stale) Mirror 19: 0x4698394e (stale) Mirror 20: 0x4698394e (stale) Mirror 21: 0x4698394e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29084: skipped 000755 1767316718.506755 read 0x00f35f thru 0x0168b1 (0x7553 bytes) 000756 1767316718.533827 read 0x010187 thru 0x01492c (0x47a6 bytes) 000757 1767316718.534434 trunc from 0x029084 to 0x00e319 (0x1ad6a bytes) 000758 1767316718.564988 mapwrite 0x01cee8 thru 0x02c4f4 (0xf60d bytes) 000762 1767316718.804985 write 0x03d297 thru 0x03ffff (0x2d69 bytes) 000764 1767316718.809748 trunc from 0x040000 to 0x00b114 (0x34eeb bytes) 000766 1767316718.853157 MIRROR_SPLIT mirror 5 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb115): Mirror 4: 0x6c80a64 (stale) Mirror 5: 0x6c80a64 (stale) Mirror 9: 0x6c80a64 (stale) Mirror 10: 0x6c80a64 (stale) Mirror 12: 0x6c80a64 (stale) Mirror 13: 0x6c80a64 (stale) Mirror 15: 0x6c80a64 (stale) Mirror 18: 0x6c80a64 (stale) Mirror 19: 0x6c80a64 (stale) Mirror 20: 0x6c80a64 (stale) Mirror 21: 0x6c80a64 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb115): Mirror 4: 0x6c80a64 (stale) Mirror 5: 0x6c80a64 (stale) Mirror 9: 0x6c80a64 (stale) Mirror 10: 0x6c80a64 (stale) Mirror 12: 0x6c80a64 (stale) Mirror 13: 0x6c80a64 (stale) Mirror 15: 0x6c80a64 (stale) Mirror 18: 0x6c80a64 (stale) Mirror 19: 0x6c80a64 (stale) Mirror 20: 0x6c80a64 (stale) Mirror 21: 0x6c80a64 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb115: skipped 000767 1767316719.316136 read 0x009b2f thru 0x00b114 (0x15e6 bytes) 000768 1767316719.321737 trunc from 0x00b115 to 0x038ece (0x2ddba bytes) 000769 1767316719.344150 write 0x01491d thru 0x01635f (0x1a43 bytes) 000772 1767316719.389110 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38ecf): Mirror 4: 0x961af47b (stale) Mirror 5: 0x961af47b (stale) Mirror 10: 0x961af47b (stale) Mirror 12: 0x961af47b (stale) Mirror 13: 0x961af47b (stale) Mirror 15: 0x961af47b (stale) Mirror 18: 0x961af47b (stale) Mirror 19: 0x961af47b (stale) Mirror 20: 0x961af47b (stale) Mirror 21: 0x961af47b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38ecf): Mirror 4: 0x961af47b (stale) Mirror 5: 0x961af47b (stale) Mirror 10: 0x961af47b (stale) Mirror 12: 0x961af47b (stale) Mirror 13: 0x961af47b (stale) Mirror 15: 0x961af47b (stale) Mirror 18: 0x961af47b (stale) Mirror 19: 0x961af47b (stale) Mirror 20: 0x961af47b (stale) Mirror 21: 0x961af47b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38ecf: skipped 000774 1767316720.005028 mapwrite 0x0057a6 thru 0x00c853 (0x70ae bytes) 000775 1767316720.088716 mapread 0x01bde7 thru 0x0285e3 (0xc7fd bytes) 000779 1767316720.110025 read 0x00a798 thru 0x018421 (0xdc8a bytes) 000781 1767316720.128151 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x38ecf): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38ecf): Mirror 4: 0x961af47b (stale) Mirror 5: 0x961af47b (stale) Mirror 10: 0x961af47b (stale) Mirror 12: 0x961af47b (stale) Mirror 13: 0x961af47b (stale) Mirror 15: 0x961af47b (stale) Mirror 18: 0x961af47b (stale) Mirror 19: 0x961af47b (stale) Mirror 20: 0x961af47b (stale) Mirror 21: 0x961af47b (stale) CRC-32 checksum value for chunk [0, 0x38ecf): Mirror 1: 0x45d0f66 Mirror 22: 0x45d0f66 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38ecf: skipped 000782 1767316720.953575 mapread 0x013025 thru 0x021b83 (0xeb5f bytes) 000783 1767316720.987394 trunc from 0x038ecf to 0x010eb4 (0x2801a bytes) 000784 1767316721.118783 mapwrite 0x02bf0c thru 0x030966 (0x4a5b bytes) 000786 1767316721.224886 mapwrite 0x03ce49 thru 0x03ffff (0x31b7 bytes) 000787 1767316721.308788 mapwrite 0x016ddd thru 0x025c1d (0xee41 bytes) 000788 1767316721.388017 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x59dfa971 (stale) Mirror 5: 0x59dfa971 (stale) Mirror 12: 0x59dfa971 (stale) Mirror 13: 0x59dfa971 (stale) Mirror 15: 0x59dfa971 (stale) Mirror 18: 0x59dfa971 (stale) Mirror 19: 0x59dfa971 (stale) Mirror 20: 0x59dfa971 (stale) Mirror 21: 0x59dfa971 (stale) Mirror 22: 0xd0e73ebe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x59dfa971 (stale) Mirror 5: 0x59dfa971 (stale) Mirror 12: 0x59dfa971 (stale) Mirror 13: 0x59dfa971 (stale) Mirror 15: 0x59dfa971 (stale) Mirror 18: 0x59dfa971 (stale) Mirror 19: 0x59dfa971 (stale) Mirror 20: 0x59dfa971 (stale) Mirror 21: 0x59dfa971 (stale) Mirror 22: 0xd0e73ebe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000790 1767316721.964180 write 0x03885d thru 0x03ffff (0x77a3 bytes) 000791 1767316721.972583 write 0x00f96a thru 0x01ea1e (0xf0b5 bytes) 000792 1767316721.984359 read 0x01eb7b thru 0x02dcaf (0xf135 bytes) 000794 1767316721.994481 mapwrite 0x03d2ff thru 0x03ffff (0x2d01 bytes) 000796 1767316722.106448 read 0x0392ef thru 0x03ffff (0x6d11 bytes) 000797 1767316722.107080 mapread 0x033372 thru 0x03b9c1 (0x8650 bytes) 000798 1767316722.118145 read 0x03f7ea thru 0x03ffff (0x816 bytes) 000799 1767316722.119162 trunc from 0x040000 to 0x032a32 (0xd5cd bytes) 000802 1767316722.142624 mapread 0x010c48 thru 0x012a60 (0x1e19 bytes) 000804 1767316722.143751 read 0x00a0c0 thru 0x00f54c (0x548d bytes) 000808 1767316722.146981 mapwrite 0x009035 thru 0x01785f (0xe82b bytes) 000810 1767316722.331621 read 0x01d602 thru 0x02947c (0xbe7b bytes) 000811 1767316722.332300 write 0x0056f5 thru 0x0150be (0xf9ca bytes) 000812 1767316722.333650 mapwrite 0x0137f0 thru 0x022706 (0xef17 bytes) 000814 1767316722.426094 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x32a33): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32a33): Mirror 4: 0xa8134008 (stale) Mirror 5: 0xa8134008 (stale) Mirror 12: 0xa8134008 (stale) Mirror 13: 0xa8134008 (stale) Mirror 15: 0xa8134008 (stale) Mirror 18: 0xa8134008 (stale) Mirror 19: 0xa8134008 (stale) Mirror 20: 0xa8134008 (stale) Mirror 21: 0xa8134008 (stale) Mirror 22: 0x7dae9192 (stale) CRC-32 checksum value for chunk [0, 0x32a33): Mirror 1: 0xc3d412aa Mirror 23: 0xc3d412aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32a33: skipped 000815 1767316723.635213 mapread 0x02e5e4 thru 0x032a32 (0x444f bytes) 000816 1767316723.664836 read 0x008870 thru 0x0168d4 (0xe065 bytes) 000818 1767316723.681765 mapwrite 0x00c1bf thru 0x013df8 (0x7c3a bytes) 000819 1767316723.884146 mapread 0x01e7e0 thru 0x028a9e (0xa2bf bytes) 000822 1767316723.904140 write 0x0085bc thru 0x01032f (0x7d74 bytes) 000824 1767316723.908338 write 0x032144 thru 0x036535 (0x43f2 bytes) 000825 1767316723.915705 trunc from 0x036536 to 0x01995f (0x1cbd6 bytes) 000828 1767316723.986373 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x19960): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19960): Mirror 4: 0x47d11d8 (stale) Mirror 5: 0x47d11d8 (stale) Mirror 12: 0x47d11d8 (stale) Mirror 13: 0x47d11d8 (stale) Mirror 15: 0x47d11d8 (stale) Mirror 18: 0x47d11d8 (stale) Mirror 19: 0x47d11d8 (stale) Mirror 20: 0x47d11d8 (stale) Mirror 21: 0x47d11d8 (stale) Mirror 22: 0xcee7c6db (stale) Mirror 23: 0x887c223e (stale) CRC-32 checksum value for chunk [0, 0x19960): Mirror 1: 0x25da7b10 Mirror 24: 0x25da7b10 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19960: skipped 000830 1767316725.126083 mapwrite 0x02af89 thru 0x02cb31 (0x1ba9 bytes) 000831 1767316725.330790 write 0x023594 thru 0x027333 (0x3da0 bytes) 000834 1767316725.339791 mapwrite 0x03ce13 thru 0x03ffff (0x31ed bytes) 000838 1767316725.432909 mapread 0x00c087 thru 0x00c4d8 (0x452 bytes) 000839 1767316725.446943 write 0x00fb99 thru 0x019f98 (0xa400 bytes) 000840 1767316725.455781 read 0x039f98 thru 0x03ffff (0x6068 bytes) 000843 1767316725.458316 mapwrite 0x00b4dd thru 0x00f64c (0x4170 bytes) 000847 1767316725.513951 mapwrite 0x033fa2 thru 0x035cf7 (0x1d56 bytes) 000848 1767316725.586464 mapread 0x03dbdd thru 0x03ec22 (0x1046 bytes) 000849 1767316725.597220 MIRROR_RESYNC 13 mirrors 000850 1767316727.614741 read 0x00f242 thru 0x012296 (0x3055 bytes) 000855 1767316727.624396 trunc from 0x040000 to 0x01018c (0x2fe73 bytes) 000856 1767316727.730407 write 0x03ad69 thru 0x03ffff (0x5297 bytes) 000857 1767316727.733010 trunc from 0x040000 to 0x017e61 (0x2819e bytes) 000858 1767316727.772000 mapwrite 0x03cb1e thru 0x03ffff (0x34e2 bytes) 000860 1767316727.874988 write 0x0188d3 thru 0x026a94 (0xe1c2 bytes) 000864 1767316727.899162 mapwrite 0x00176f thru 0x00237b (0xc0d bytes) 000865 1767316727.960747 MIRROR_RESYNC 13 mirrors 000866 1767316730.233038 read 0x03b902 thru 0x03ffff (0x46fe bytes) 000867 1767316730.244337 read 0x0017a4 thru 0x004375 (0x2bd2 bytes) 000869 1767316730.263412 write 0x00fcfa thru 0x01d6ac (0xd9b3 bytes) 000870 1767316730.362534 mapwrite 0x03b5b1 thru 0x03ffff (0x4a4f bytes) 000871 1767316730.472315 MIRROR_RESYNC 13 mirrors 000873 1767316732.395466 mapwrite 0x0040bf thru 0x006b5e (0x2aa0 bytes) 000874 1767316732.562558 read 0x00bb61 thru 0x017f90 (0xc430 bytes) 000877 1767316732.577322 mapwrite 0x03131d thru 0x0329c0 (0x16a4 bytes) 000878 1767316732.652077 MIRROR_SPLIT mirror 8 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x92f4c267 (stale) Mirror 5: 0x92f4c267 (stale) Mirror 12: 0x92f4c267 (stale) Mirror 13: 0x92f4c267 (stale) Mirror 15: 0x92f4c267 (stale) Mirror 18: 0x92f4c267 (stale) Mirror 19: 0x92f4c267 (stale) Mirror 21: 0x92f4c267 (stale) Mirror 22: 0x92f4c267 (stale) Mirror 23: 0x92f4c267 (stale) Mirror 24: 0x92f4c267 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x92f4c267 (stale) Mirror 5: 0x92f4c267 (stale) Mirror 12: 0x92f4c267 (stale) Mirror 13: 0x92f4c267 (stale) Mirror 15: 0x92f4c267 (stale) Mirror 18: 0x92f4c267 (stale) Mirror 19: 0x92f4c267 (stale) Mirror 21: 0x92f4c267 (stale) Mirror 22: 0x92f4c267 (stale) Mirror 23: 0x92f4c267 (stale) Mirror 24: 0x92f4c267 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000879 1767316733.381535 mapread 0x034fe1 thru 0x03ffff (0xb01f bytes) 000881 1767316733.414059 read 0x026429 thru 0x02e61c (0x81f4 bytes) 000883 1767316733.428635 read 0x0097bc thru 0x00d7a9 (0x3fee bytes) 000884 1767316733.441449 trunc from 0x040000 to 0x021d39 (0x1e2c6 bytes) 000886 1767316733.463415 mapwrite 0x006e67 thru 0x00a3f6 (0x3590 bytes) 000887 1767316733.562615 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x21d3a): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21d3a): Mirror 4: 0xa2e72cf4 (stale) Mirror 5: 0xa2e72cf4 (stale) Mirror 12: 0xa2e72cf4 (stale) Mirror 13: 0xa2e72cf4 (stale) Mirror 15: 0xa2e72cf4 (stale) Mirror 18: 0xa2e72cf4 (stale) Mirror 19: 0xa2e72cf4 (stale) Mirror 21: 0xa2e72cf4 (stale) Mirror 22: 0xa2e72cf4 (stale) Mirror 23: 0xa2e72cf4 (stale) Mirror 24: 0xa2e72cf4 (stale) CRC-32 checksum value for chunk [0, 0x21d3a): Mirror 1: 0x59f7ee7e Mirror 25: 0x59f7ee7e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21d3a: skipped 000888 1767316734.511582 mapwrite 0x028e3e thru 0x02e332 (0x54f5 bytes) 000892 1767316734.722270 mapwrite 0x021c2e thru 0x023237 (0x160a bytes) 000896 1767316734.810482 mapread 0x027276 thru 0x02e332 (0x70bd bytes) 000898 1767316734.812697 mapread 0x016466 thru 0x023caa (0xd845 bytes) 000899 1767316734.831731 write 0x03cd99 thru 0x03ffff (0x3267 bytes) 000900 1767316734.833470 read 0x031885 thru 0x03b720 (0x9e9c bytes) 000901 1767316734.849986 trunc from 0x040000 to 0x01e2c0 (0x21d3f bytes) 000902 1767316734.883758 mapwrite 0x002f8d thru 0x006dda (0x3e4e bytes) 000904 1767316734.983292 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x1e2c1): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e2c1): Mirror 4: 0x333e2552 (stale) Mirror 5: 0x333e2552 (stale) Mirror 12: 0x333e2552 (stale) Mirror 13: 0x333e2552 (stale) Mirror 15: 0x333e2552 (stale) Mirror 18: 0x333e2552 (stale) Mirror 19: 0x333e2552 (stale) Mirror 21: 0x333e2552 (stale) Mirror 22: 0x333e2552 (stale) Mirror 23: 0x333e2552 (stale) Mirror 24: 0x333e2552 (stale) Mirror 25: 0xda80c3a9 (stale) CRC-32 checksum value for chunk [0, 0x1e2c1): Mirror 1: 0x40a3143c Mirror 26: 0x40a3143c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e2c1: skipped 000906 1767316736.081706 read 0x0049c9 thru 0x010f47 (0xc57f bytes) 000907 1767316736.098178 mapread 0x0013db thru 0x00dcb0 (0xc8d6 bytes) 000908 1767316736.133896 mapread 0x00dfac thru 0x01decd (0xff22 bytes) 000911 1767316736.191527 read 0x018887 thru 0x01e2c0 (0x5a3a bytes) 000912 1767316736.204184 MIRROR_SPLIT mirror 13 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e2c1): Mirror 4: 0x333e2552 (stale) Mirror 5: 0x333e2552 (stale) Mirror 12: 0x333e2552 (stale) Mirror 13: 0x333e2552 (stale) Mirror 15: 0x333e2552 (stale) Mirror 18: 0x333e2552 (stale) Mirror 19: 0x333e2552 (stale) Mirror 21: 0x333e2552 (stale) Mirror 22: 0x333e2552 (stale) Mirror 23: 0x333e2552 (stale) Mirror 24: 0x333e2552 (stale) Mirror 25: 0xda80c3a9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e2c1): Mirror 4: 0x333e2552 (stale) Mirror 5: 0x333e2552 (stale) Mirror 12: 0x333e2552 (stale) Mirror 13: 0x333e2552 (stale) Mirror 15: 0x333e2552 (stale) Mirror 18: 0x333e2552 (stale) Mirror 19: 0x333e2552 (stale) Mirror 21: 0x333e2552 (stale) Mirror 22: 0x333e2552 (stale) Mirror 23: 0x333e2552 (stale) Mirror 24: 0x333e2552 (stale) Mirror 25: 0xda80c3a9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e2c1: skipped truncating to largest ever: 0x3db02 000915 1767316736.756449 trunc from 0x01e2c1 to 0x03db01 (0x1f841 bytes) 000917 1767316736.792808 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x3db02): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3db02): Mirror 4: 0xfe794d (stale) Mirror 5: 0xfe794d (stale) Mirror 12: 0xfe794d (stale) Mirror 13: 0xfe794d (stale) Mirror 15: 0xfe794d (stale) Mirror 18: 0xfe794d (stale) Mirror 19: 0xfe794d (stale) Mirror 21: 0xfe794d (stale) Mirror 22: 0xfe794d (stale) Mirror 23: 0xfe794d (stale) Mirror 24: 0xfe794d (stale) Mirror 25: 0x7b038212 (stale) CRC-32 checksum value for chunk [0, 0x3db02): Mirror 1: 0x54657376 Mirror 26: 0x54657376 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3db02: skipped 000918 1767316737.788807 mapwrite 0x00d953 thru 0x0164d4 (0x8b82 bytes) 000920 1767316737.987443 read 0x03cf2d thru 0x03db01 (0xbd5 bytes) 000921 1767316737.993361 mapwrite 0x022fdb thru 0x0281a8 (0x51ce bytes) 000922 1767316738.047211 mapread 0x00ba03 thru 0x01123d (0x583b bytes) 000927 1767316738.051975 trunc from 0x03db02 to 0x03bc67 (0x1e9a bytes) 000929 1767316738.076994 mapread 0x023a6e thru 0x0296b0 (0x5c43 bytes) 000930 1767316738.079333 mapwrite 0x028525 thru 0x037933 (0xf40f bytes) 000933 1767316738.178057 mapwrite 0x000ae3 thru 0x00600a (0x5528 bytes) 000934 1767316738.235372 read 0x038db7 thru 0x03b39e (0x25e8 bytes) 000935 1767316738.236557 trunc from 0x03bc68 to 0x01fbe4 (0x1c083 bytes) 000938 1767316738.256065 write 0x01b6a0 thru 0x01ea58 (0x33b9 bytes) 000943 1767316738.260750 mapwrite 0x03c223 thru 0x03ffff (0x3ddd bytes) 000944 1767316738.337756 mapwrite 0x01f072 thru 0x024724 (0x56b3 bytes) 000945 1767316738.385237 trunc from 0x040000 to 0x01eefa (0x21105 bytes) truncating to largest ever: 0x3dfbf 000947 1767316738.406524 trunc from 0x01eefb to 0x03dfbe (0x1f0c4 bytes) 000949 1767316738.432152 write 0x00674d thru 0x00ee49 (0x86fd bytes) 000951 1767316738.434426 mapwrite 0x01cd9c thru 0x02858c (0xb7f1 bytes) 000953 1767316738.607739 write 0x030c45 thru 0x03402e (0x33ea bytes) 000954 1767316738.630037 write 0x005480 thru 0x00f690 (0xa211 bytes) 000956 1767316738.631890 trunc from 0x03dfbf to 0x03b26c (0x2d52 bytes) 000957 1767316738.707217 read 0x011cbe thru 0x01effd (0xd340 bytes) 000960 1767316738.725802 write 0x0245aa thru 0x031749 (0xd1a0 bytes) 000961 1767316738.749387 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x3b26d): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b26d): Mirror 4: 0x19fc3584 (stale) Mirror 5: 0x19fc3584 (stale) Mirror 12: 0x19fc3584 (stale) Mirror 13: 0x19fc3584 (stale) Mirror 15: 0x19fc3584 (stale) Mirror 18: 0x19fc3584 (stale) Mirror 19: 0x19fc3584 (stale) Mirror 21: 0x19fc3584 (stale) Mirror 22: 0x19fc3584 (stale) Mirror 23: 0x19fc3584 (stale) Mirror 24: 0x19fc3584 (stale) Mirror 25: 0xaf32bd0e (stale) Mirror 26: 0x840e3709 (stale) CRC-32 checksum value for chunk [0, 0x3b26d): Mirror 1: 0x9d0330ef Mirror 27: 0x9d0330ef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b26d: skipped 000962 1767316740.244212 write 0x01311c thru 0x022563 (0xf448 bytes) 000964 1767316740.386824 MIRROR_RESYNC 15 mirrors 000965 1767316742.766516 write 0x0093b7 thru 0x009733 (0x37d bytes) 000966 1767316742.834321 mapread 0x0251e0 thru 0x02a55c (0x537d bytes) 000969 1767316742.850388 write 0x03abf1 thru 0x03ffff (0x540f bytes) 000971 1767316742.859838 trunc from 0x040000 to 0x03a8af (0x5750 bytes) 000972 1767316742.887185 mapwrite 0x032597 thru 0x0327b9 (0x223 bytes) 000973 1767316742.948557 trunc from 0x03a8b0 to 0x01a2ac (0x20603 bytes) 000977 1767316742.967077 trunc from 0x01a2ad to 0x020764 (0x64b8 bytes) 000978 1767316742.992408 trunc from 0x020765 to 0x01a8ca (0x5e9a bytes) 000980 1767316743.014113 trunc from 0x01a8cb to 0x0305be (0x15cf4 bytes) 000981 1767316743.046050 mapread 0x02614b thru 0x0305be (0xa474 bytes) 000982 1767316743.071446 MIRROR_SPLIT mirror 6 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x305bf): Mirror 4: 0x14d9eb00 (stale) Mirror 5: 0x14d9eb00 (stale) Mirror 12: 0x14d9eb00 (stale) Mirror 13: 0x14d9eb00 (stale) Mirror 15: 0x14d9eb00 (stale) Mirror 19: 0x14d9eb00 (stale) Mirror 21: 0x14d9eb00 (stale) Mirror 22: 0x14d9eb00 (stale) Mirror 23: 0x14d9eb00 (stale) Mirror 24: 0x14d9eb00 (stale) Mirror 25: 0x14d9eb00 (stale) Mirror 26: 0x14d9eb00 (stale) Mirror 27: 0x14d9eb00 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x305bf): Mirror 4: 0x14d9eb00 (stale) Mirror 5: 0x14d9eb00 (stale) Mirror 12: 0x14d9eb00 (stale) Mirror 13: 0x14d9eb00 (stale) Mirror 15: 0x14d9eb00 (stale) Mirror 19: 0x14d9eb00 (stale) Mirror 21: 0x14d9eb00 (stale) Mirror 22: 0x14d9eb00 (stale) Mirror 23: 0x14d9eb00 (stale) Mirror 24: 0x14d9eb00 (stale) Mirror 25: 0x14d9eb00 (stale) Mirror 26: 0x14d9eb00 (stale) Mirror 27: 0x14d9eb00 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x305bf: skipped 000983 1767316743.708179 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x305bf): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x305bf): Mirror 4: 0x14d9eb00 (stale) Mirror 5: 0x14d9eb00 (stale) Mirror 12: 0x14d9eb00 (stale) Mirror 13: 0x14d9eb00 (stale) Mirror 15: 0x14d9eb00 (stale) Mirror 19: 0x14d9eb00 (stale) Mirror 21: 0x14d9eb00 (stale) Mirror 22: 0x14d9eb00 (stale) Mirror 23: 0x14d9eb00 (stale) Mirror 24: 0x14d9eb00 (stale) Mirror 25: 0x14d9eb00 (stale) Mirror 26: 0x14d9eb00 (stale) Mirror 27: 0x14d9eb00 (stale) CRC-32 checksum value for chunk [0, 0x305bf): Mirror 1: 0x2fb902bc Mirror 28: 0x2fb902bc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x305bf: skipped 000984 1767316744.821384 trunc from 0x0305bf to 0x009dda (0x267e4 bytes) 000986 1767316744.971541 trunc from 0x009ddb to 0x03dc9c (0x33ec2 bytes) 000987 1767316744.991673 read 0x02bd41 thru 0x0337c5 (0x7a85 bytes) 000988 1767316745.009584 MIRROR_RESYNC 15 mirrors 000989 1767316747.167436 MIRROR_SPLIT mirror 13 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) 000990 1767316747.843088 MIRROR_RESYNC 14 mirrors 000991 1767316748.540791 write 0x009a01 thru 0x0145a5 (0xaba5 bytes) 000992 1767316748.629003 mapread 0x025b1c thru 0x027d9e (0x2283 bytes) 000993 1767316748.644277 MIRROR_SPLIT mirror 2 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dc9d): Mirror 4: 0x9f68c8ab (stale) Mirror 12: 0x9f68c8ab (stale) Mirror 13: 0x9f68c8ab (stale) Mirror 15: 0x9f68c8ab (stale) Mirror 19: 0x9f68c8ab (stale) Mirror 21: 0x9f68c8ab (stale) Mirror 22: 0x9f68c8ab (stale) Mirror 23: 0x9f68c8ab (stale) Mirror 24: 0x9f68c8ab (stale) Mirror 25: 0x9f68c8ab (stale) Mirror 26: 0x9f68c8ab (stale) Mirror 28: 0x9f68c8ab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc9d): Mirror 4: 0x9f68c8ab (stale) Mirror 12: 0x9f68c8ab (stale) Mirror 13: 0x9f68c8ab (stale) Mirror 15: 0x9f68c8ab (stale) Mirror 19: 0x9f68c8ab (stale) Mirror 21: 0x9f68c8ab (stale) Mirror 22: 0x9f68c8ab (stale) Mirror 23: 0x9f68c8ab (stale) Mirror 24: 0x9f68c8ab (stale) Mirror 25: 0x9f68c8ab (stale) Mirror 26: 0x9f68c8ab (stale) Mirror 28: 0x9f68c8ab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc9d: skipped 000995 1767316749.293403 mapwrite 0x0113a3 thru 0x01b44e (0xa0ac bytes) 000997 1767316749.375669 read 0x00e1ac thru 0x01638a (0x81df bytes) 000999 1767316749.376700 read 0x0373ec thru 0x038b54 (0x1769 bytes) 001000 1767316749.389350 mapwrite 0x02915b thru 0x02e52d (0x53d3 bytes) All operations completed A-OK! PASS 70a (115s) == sanity-flr test 71: check mirror extend parent fid ==== 20:19:14 (1767316754) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 20:19:17 (1767316757) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x259:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.175471 s, 12.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x259:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15151 s, 6.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25a:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x259:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25a:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x259:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25f:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 20:19:28 (1767316768) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre2 Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8423776 .. Extending file size to 6549760 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6874144 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8510592 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8487008 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8946752 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8823424 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8986912 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9164992 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9113600 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9186688 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57910 57912 57913 57916 57917 failed 192.168.204.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg430-client.virtnet /mnt/lustre2 (opts:) 192.168.204.130@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg430-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:20:47 (1767316847) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre2 Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4884896 .. Extending file size to 4775296 .. failed Extending file size to 8839808 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6822656 .. Extending file size to 8290944 .. done Extending file size to 9090720 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8347872 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8948928 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59663 59665 59666 59668 59670 done 192.168.204.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg430-client.virtnet /mnt/lustre2 (opts:) 192.168.204.130@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg430-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:22:10 (1767316930) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4224 2204288 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 25600 3743744 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4224 2204288 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 25600 3743744 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.13548 s, 15.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.301074 s, 7.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xd7f15b17 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg430-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:22:28 (1767316948) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0710896 s, 29.5 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 20:22:34 (1767316954) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f3:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x270:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x270:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x271:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x274:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:22:40 (1767316960) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.180779 s, 11.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x272:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x275:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x276:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:22:47 (1767316967) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.316879 s, 6.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x273:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x274:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x277:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x273:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x274:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x277:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x278:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:22:55 (1767316975) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.202392 s, 10.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x275:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x279:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x275:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x279:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27a:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:23:28 (1767317008) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg430-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870976 s, 12.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x276:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x276:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x277:0x0] } PASS 204d (58s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:24:26 (1767317066) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg430-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x278:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x279:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.395253 s, 5.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x278:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x279:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x278:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x279:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (61s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:25:27 (1767317127) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg430-server mds-ost sync done. Creating new pool test_204f oleg430-server: Pool lustre.test_204f created Adding targets to pool oleg430-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg430-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.433911 s, 4.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x27a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x27a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg430-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg430-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg430-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:26:48 (1767317208) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:26:54 (1767317214) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:27:01 (1767317221) oleg430-server: Pool lustre.test_206 created Adding targets to pool oleg430-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg430-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Updated after 2s: want 'lustre-OST0000_UUID lustre-OST0001_UUID ' got 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg430-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg430-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg430-server: Pool lustre.test_206 destroyed PASS 206 (24s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:27:25 (1767317245) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609696 s, 17.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403362 s, 26.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x440639be Mirror 3: 0x440639be PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:27:35 (1767317255) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:27:38 (1767317258) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:27:41 (1767317261) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.148505 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1480000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1120000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1120000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1120000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x21f0000 Read this (4 bytes): BBBB 192.168.204.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg430-client.virtnet /mnt/lustre2 (opts:) PASS 209a (32s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:28:13 (1767317293) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0109203 s, 750 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:28:20 (1767317300) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476208 s, 22.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x218 lmm_fid: [0x200000404:0x218:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x218 lmm_fid: [0x200000404:0x218:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x289:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834456 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-01 20:28:22.000000000 -0500 Modify: 2026-01-01 20:28:22.000000000 -0500 Change: 2026-01-01 20:28:23.000000000 -0500 Birth: 2026-01-01 20:28:23.000000000 -0500 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:28:28 (1767317308) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:28:31 (1767317311) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.503783 s, 81.3 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.029494 s, 139 kB/s size after second write -rw-r--r-- 1 root root 4096 Jan 1 20:28 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21a:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jan 1 20:28 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:28:39 (1767317319) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:28:47 (1767317327) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1874 sec =========== 20:28:56 (1767317336) === sanity-flr: start cleanup 20:28:58 (1767317338) === === sanity-flr: finish cleanup 20:29:04 (1767317344) ===