-----============= acceptance-small: sanity-flr ============----- Thu Jan 1 22:05:32 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg228-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 22:05:49 (1767323149) === oleg228-client.virtnet: executing check_config_client /mnt/lustre oleg228-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg228-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff89d686b83000.idle_timeout=debug osc.lustre-OST0001-osc-ffff89d686b83000.idle_timeout=debug disable quota as required oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 22:06:34 (1767323194) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f10289] == sanity-flr test 0a: lfs mirror create with -N option == 22:06:36 (1767323196) PASS 0a (9s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 22:06:45 (1767323205) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 22:06:47 (1767323207) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 22:06:50 (1767323210) PASS 0d (15s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 22:07:04 (1767323224) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 22:07:06 (1767323226) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 22:07:09 (1767323229) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff89d686b83000.stats=clear osc.lustre-OST0001-osc-ffff89d686b83000.stats=clear PASS 0g (5s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 22:07:14 (1767323234) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (6s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 22:07:20 (1767323240) PASS 0j (6s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 22:07:26 (1767323246) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 22:07:28 (1767323248) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (6s) == sanity-flr test 2: create components from existing files ========================================================== 22:07:34 (1767323254) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 22:07:38 (1767323258) PASS 3 (5s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 22:07:43 (1767323263) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 22:07:48 (1767323268) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (4s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 22:07:54 (1767323274) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 22:07:57 (1767323277) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.412266 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060104 s, 17.4 MB/s PASS 21 (6s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 22:08:03 (1767323283) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0471011 s, 44.5 MB/s PASS 22 (5s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 22:08:08 (1767323288) Stopping /mnt/lustre-ost1 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89d686b83000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89d686b83000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg228-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89d686b83000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89d686b83000.ost_server_uuid in DISCONN state after 1 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg228-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (55s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:09:03 (1767323343) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.414721 s, 22.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89d686b83000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89d686b83000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg228-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (27s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:09:30 (1767323370) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89d686b83000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89d686b83000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg228-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89d686b83000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89d686b83000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg228-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (53s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 22:10:23 (1767323423) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.91649 s, 27.4 MB/s -rw-r--r-- 1 root root 52428800 Jan 1 22:10 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.645s user 0m0.001s sys 0m0.156s Stopping /mnt/lustre-ost1 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89d686b83000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89d686b83000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.540s user 0m0.000s sys 0m0.150s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg228-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89d686b83000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89d686b83000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.688s user 0m0.000s sys 0m0.175s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg228-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg228-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89d686b83000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89d686b83000.ost_server_uuid in FULL state after 0 sec PASS 33b (55s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 22:11:18 (1767323478) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 22:11:20 (1767323480) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 22:11:23 (1767323483) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 22:11:25 (1767323485) PASS 35 (4s) == sanity-flr test 36a: write to mirrored files ========== 22:11:29 (1767323489) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.337292 s, 37.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (8s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 22:11:37 (1767323497) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243765 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297688 s, 35.2 MB/s PASS 36b (4s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 22:11:41 (1767323501) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 22:11:46 (1767323506) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00104958 s, 976 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000784255 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000655074 s, 1.6 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000994356 s, 1.0 MB/s flv=10 olv=10 PASS 36d (7s) == sanity-flr test 37: mirror I/O API verification ======= 22:11:53 (1767323513) create test files with size 15745781 .. 48aa328c9b25528334c10c60e72a14c6 - 0016171908aadf2097271a272376ffd9 - f0896badcd19e60e87aa52e8114bf171 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre PASS 37 (13s) == sanity-flr test 38: resync ============================ 22:12:06 (1767323526) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (7s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 22:12:13 (1767323533) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:12:16 (1767323536) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459591 s, 22.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3a:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x39:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273241 s, 38.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (5s) == sanity-flr test 41: lfs mirror resync check =========== 22:12:21 (1767323541) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0718355 s, 29.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.204556 s, 20.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.127707 s, 16.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (10s) == sanity-flr test 42: lfs mirror verify ================= 22:12:31 (1767323551) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 22:12:33 (1767323553) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:12:35 (1767323555) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176036 s, 59.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (5s) == sanity-flr test 44a: lfs mirror split check =========== 22:12:40 (1767323560) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.259204 s, 12.1 MB/s PASS 44a (7s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:12:47 (1767323567) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x46:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x46:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x46:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (5s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:12:53 (1767323573) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.228221 s, 45.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:13:02 (1767323582) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff89d699962800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.210589 s, 49.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff89d687298000.xattr_cache=1 PASS 44e (10s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:13:12 (1767323592) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (24s) == sanity-flr test 46: Verify setstripe --copy option ==== 22:13:37 (1767323617) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (22s) == sanity-flr test 47: Verify mirror obj alloc =========== 22:13:58 (1767323638) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 22:14:00 (1767323640) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.137792 s, 22.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0659461 s, 47.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xfcf049f8 (stale) Mirror 2: 0xfcf049f8 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 79dad4bdab8e398ae91089f6c5555551 - mirror 1 checksum: 067be9668bf3b9b2c592436909fa7ee0 - mirror 2 checksum: 79dad4bdab8e398ae91089f6c5555551 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jan 1 22:14:02 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jan 1 22:14:02 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 79dad4bdab8e398ae91089f6c5555551 - mirror 1 checksum: 067be9668bf3b9b2c592436909fa7ee0 - mirror 2 checksum: 067be9668bf3b9b2c592436909fa7ee0 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jan 1 22:14:02 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jan 1 22:14:02 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } PASS 48 (5s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 22:14:05 (1767323645) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01226 s, 85.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 404480.. 405503: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (4s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 22:14:09 (1767323649) Failing mds1 on oleg228-server Stopping /mnt/lustre-mds1 (opts:) on oleg228-server 22:14:18 (1767323658) shut down facet: mds1 facet_host: oleg228-server facet_failover_host: oleg228-server Failover mds1 to oleg228-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-MDT0000 22:14:40 (1767323680) targets are mounted 22:14:40 (1767323680) facet_failover done oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278769 s, 37.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } PASS 50A (40s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 22:14:49 (1767323689) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0383714 s, 534 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0111574 s, 1.8 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (5s) == sanity-flr test 50b: mirror rsync handles sparseness == 22:14:54 (1767323694) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0255835 s, 801 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00598164 s, 3.4 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (6s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 22:15:00 (1767323700) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00428792 s, 3.8 MB/s PASS 50c (4s) == sanity-flr test 50d: mirror rsync keep holes ========== 22:15:04 (1767323704) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0392916 s, 521 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 22:15:08 (1767323708) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00387061 s, 1.1 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0373767 s, 110 kB/s PASS 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 22:15:12 (1767323712) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) PASS 60b (12s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 22:15:24 (1767323724) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (22s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 22:15:46 (1767323746) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (20s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 22:16:06 (1767323766) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (20s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 22:16:26 (1767323786) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (4s) == sanity-flr test 70: mirror create and split race ====== 22:16:30 (1767323790) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 58547 mirror split pid 58548 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4952: 58547 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4952: 58548 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (64s) == sanity-flr test 70a: flr mode fsx test ================ 22:17:34 (1767323854) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 3855 skipping zero size read fallocating to largest ever: 0x23c0c 000002 1767323856.164680 fallocate 0x01bb49 thru 0x023c0b (0x80c3 bytes) 000003 1767323856.168504 mapread 0x00828e thru 0x00a031 (0x1da4 bytes) 000006 1767323856.185959 trunc from 0x02336b to 0x023c0b (0x8a1 bytes) 000007 1767323856.190430 trunc from 0x016da8 to 0x016ea5 (0xfe bytes) 000008 1767323856.194123 mapwrite 0x01ab62 thru 0x029f5c (0xf3fb bytes) 000009 1767323856.242644 write 0x0342b9 thru 0x03cd17 (0x8a5f bytes) 000010 1767323856.243983 trunc from 0x03cd18 to 0x0079cc (0x3534b bytes) 000012 1767323856.259959 trunc from 0x004d57 to 0x0079cc (0x2c76 bytes) truncating to largest ever: 0x2f30c 000013 1767323856.266860 trunc from 0x0079cd to 0x02f30b (0x2793f bytes) 000014 1767323856.277848 punch from 0x01d95f to 0x023dfb (0x649d bytes) 000015 1767323856.281834 mapread 0x009e61 thru 0x00b0d4 (0x1274 bytes) 000016 1767323856.288888 read 0x027b3e thru 0x02f30b (0x77ce bytes) 000017 1767323856.297437 write 0x026618 thru 0x0295be (0x2fa7 bytes) 000018 1767323856.305154 mapread 0x00d094 thru 0x00eadf (0x1a4c bytes) 000019 1767323856.307088 trunc from 0x012374 to 0x0148be (0x254b bytes) 000020 1767323856.311701 trunc from 0x02cc81 to 0x02f30b (0x268b bytes) 000021 1767323856.318983 MIRROR_EXTEND to 2 mirrors 000022 1767323856.763103 trunc from 0x02f30c to 0x00efb5 (0x20356 bytes) 000023 1767323856.833205 trunc from 0x00efb6 to 0x01e5ef (0xf63a bytes) 000024 1767323856.852791 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x303:0x0]: Invalid argument (22) 000025 1767323856.966291 read 0x007572 thru 0x0121c1 (0xac50 bytes) 000026 1767323856.973733 trunc from 0x005c89 to 0x015617 (0xf98f bytes) 000028 1767323856.978385 read 0x004d47 thru 0x01342c (0xe6e6 bytes) 000032 1767323856.988674 mapread 0x00bf18 thru 0x010c66 (0x4d4f bytes) 000033 1767323856.989674 fallocate 0x01532f thru 0x01c375 (0x7047 bytes) fallocating to largest ever: 0x3ce85 000034 1767323856.992985 fallocate 0x02ee3b thru 0x03ce84 (0xe04a bytes) 000035 1767323856.996728 read 0x0066a7 thru 0x00882b (0x2185 bytes) 000036 1767323857.000739 MIRROR_EXTEND to 2 mirrors 000037 1767323857.446133 trunc from 0x03ce85 to 0x024c44 (0x18240 bytes) 000038 1767323857.487034 mapwrite 0x02af82 thru 0x03403c (0x90bb bytes) 000042 1767323857.536499 trunc from 0x03403d to 0x0326ff (0x193d bytes) 000044 1767323857.551156 trunc from 0x032700 to 0x01bf3f (0x167c0 bytes) 000045 1767323857.569116 read 0x00dab8 thru 0x00e5ce (0xb17 bytes) 000047 1767323857.581739 trunc from 0x01bf40 to 0x009a8e (0x124b1 bytes) 000049 1767323857.602809 trunc from 0x005c7d to 0x009a8e (0x3e12 bytes) 000050 1767323857.610291 fallocate 0x020f1b thru 0x024284 (0x336a bytes) 000051 1767323857.621913 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x303:0x0]: Invalid argument (22) 000052 1767323857.756671 trunc from 0x01b5d3 to 0x024284 (0x8cb2 bytes) 000053 1767323857.760743 mapwrite 0x008a66 thru 0x009c69 (0x1204 bytes) 000054 1767323857.780009 trunc from 0x01da6a to 0x024284 (0x681b bytes) 000055 1767323857.783396 write 0x0101bb thru 0x010523 (0x369 bytes) 000056 1767323857.796656 mapwrite 0x03b82d thru 0x03ffff (0x47d3 bytes) 000057 1767323857.833315 trunc from 0x008269 to 0x009f02 (0x1c9a bytes) 000058 1767323857.836290 mapread 0x0329b1 thru 0x0365d7 (0x3c27 bytes) 000059 1767323857.845585 read 0x00c121 thru 0x019cd8 (0xdbb8 bytes) 000060 1767323857.852310 mapwrite 0x03d50c thru 0x03ffff (0x2af4 bytes) 000061 1767323857.864804 trunc from 0x02b57b to 0x037f5a (0xc9e0 bytes) 000062 1767323857.869729 write 0x017b9e thru 0x01ab71 (0x2fd4 bytes) 000063 1767323857.874013 mapwrite 0x029111 thru 0x038c2a (0xfb1a bytes) zero_range to largest ever: 0x40000 000066 1767323857.903124 trunc from 0x03c33f to 0x03ffff (0x3cc1 bytes) 000067 1767323857.908320 write 0x02308e thru 0x031aae (0xea21 bytes) 000068 1767323857.909317 trunc from 0x040000 to 0x0214ac (0x1eb53 bytes) 000069 1767323857.926890 write 0x022534 thru 0x02776a (0x5237 bytes) 000070 1767323857.928966 mapread 0x0081a0 thru 0x010db5 (0x8c16 bytes) 000071 1767323857.937245 fallocate 0x02ea92 thru 0x03c22e (0xd79d bytes) 000072 1767323857.947338 trunc from 0x01b470 to 0x0209d5 (0x5566 bytes) 000073 1767323857.952830 mapwrite 0x007ead thru 0x00f640 (0x7794 bytes) 000075 1767323857.975077 trunc from 0x01e552 to 0x027d4a (0x97f9 bytes) 000076 1767323857.982242 trunc from 0x01b0be to 0x01bc54 (0xb97 bytes) 000077 1767323857.987620 mapwrite 0x0356f5 thru 0x03ffff (0xa90b bytes) 000078 1767323858.037694 read 0x039245 thru 0x03e4bb (0x5277 bytes) 000079 1767323858.038766 mapwrite 0x02a2d0 thru 0x037b6f (0xd8a0 bytes) 000080 1767323858.077214 write 0x00b64d thru 0x00b84e (0x202 bytes) 000082 1767323858.079360 punch from 0x025264 to 0x02acca (0x5a67 bytes) 000083 1767323858.082770 mapread 0x02dbb2 thru 0x02e011 (0x460 bytes) 000085 1767323858.083971 write 0x0173ee thru 0x018684 (0x1297 bytes) 000086 1767323858.084802 trunc from 0x040000 to 0x0374f0 (0x8b0f bytes) 000087 1767323858.095359 punch from 0x005e13 to 0x007043 (0x1231 bytes) 000088 1767323858.100999 punch from 0x003ddc to 0x00b6f6 (0x791b bytes) 000090 1767323858.113165 read 0x004e40 thru 0x0070d1 (0x2292 bytes) 000092 1767323858.121819 punch from 0x00c58a to 0x01280e (0x6285 bytes) 000093 1767323858.125863 read 0x018997 thru 0x01f1cd (0x6837 bytes) 000094 1767323858.126941 trunc from 0x0374f1 to 0x031c36 (0x58ba bytes) 000095 1767323858.136192 fallocate 0x030e40 thru 0x0368b0 (0x5a71 bytes) 000096 1767323858.139332 mapwrite 0x00981c thru 0x015e85 (0xc66a bytes) 000097 1767323858.159223 trunc from 0x0368b1 to 0x01354a (0x23366 bytes) 000098 1767323858.173961 read 0x004689 thru 0x007810 (0x3188 bytes) 000099 1767323858.175227 read 0x006015 thru 0x010ee1 (0xaecd bytes) 000100 1767323858.175844 write 0x0038c3 thru 0x0079f1 (0x412f bytes) 000101 1767323858.177417 read 0x00261a thru 0x0114c8 (0xeeaf bytes) 000102 1767323858.178128 mapwrite 0x001aba thru 0x00a4bb (0x8a02 bytes) 000103 1767323858.194639 read 0x002a50 thru 0x00bf32 (0x94e3 bytes) 000104 1767323858.195238 read 0x004c7c thru 0x00568f (0xa14 bytes) 000106 1767323858.197500 read 0x009e84 thru 0x01342c (0x95a9 bytes) 000107 1767323858.198607 trunc from 0x00b24d to 0x00cd7b (0x1b2f bytes) 000108 1767323858.205911 mapwrite 0x01bd6c thru 0x0259cd (0x9c62 bytes) 000109 1767323858.240344 fallocate 0x010fc3 thru 0x015227 (0x4265 bytes) 000110 1767323858.245896 mapread 0x01627b thru 0x01c420 (0x61a6 bytes) 000111 1767323858.247890 punch from 0x0091b1 to 0x00c641 (0x3491 bytes) 000112 1767323858.251920 fallocate 0x007a22 thru 0x01198a (0x9f69 bytes) 000114 1767323858.256292 punch from 0x0108ca to 0x0111e7 (0x91e bytes) 000116 1767323858.261594 fallocate 0x001dc9 thru 0x010264 (0xe49c bytes) 000118 1767323858.267050 fallocate 0x039c95 thru 0x03ffff (0x636b bytes) 000119 1767323858.271460 write 0x02b23d thru 0x02c3a4 (0x1168 bytes) 000120 1767323858.273179 punch from 0x015415 to 0x01fc59 (0xa845 bytes) 000121 1767323858.277760 punch from 0x02a205 to 0x02c3a4 (0x21a0 bytes) 000122 1767323858.286219 trunc from 0x022c34 to 0x02b1a5 (0x8572 bytes) 000123 1767323858.289798 mapwrite 0x015836 thru 0x01a71d (0x4ee8 bytes) 000125 1767323858.312169 trunc from 0x02c3a5 to 0x030ec5 (0x4b21 bytes) 000126 1767323858.318987 trunc from 0x023bb8 to 0x02ba5f (0x7ea8 bytes) 000129 1767323858.322212 mapread 0x01c4de thru 0x023171 (0x6c94 bytes) 000130 1767323858.340140 trunc from 0x030ec6 to 0x00dd4a (0x2317b bytes) 000132 1767323858.361742 read 0x0021a8 thru 0x005321 (0x317a bytes) 000133 1767323858.362444 mapwrite 0x01e803 thru 0x028294 (0x9a92 bytes) 000134 1767323858.409061 punch from 0x011603 to 0x012bc1 (0x15bf bytes) 000135 1767323858.412088 trunc from 0x012ff7 to 0x01e995 (0xb99f bytes) 000136 1767323858.415690 write 0x022980 thru 0x02de07 (0xb488 bytes) 000137 1767323858.417128 trunc from 0x027958 to 0x02de07 (0x64b0 bytes) 000138 1767323858.425194 mapread 0x01796b thru 0x01fee7 (0x857d bytes) 000140 1767323858.473225 fallocate 0x010880 thru 0x01a0e4 (0x9865 bytes) 000141 1767323858.487380 trunc from 0x0066a7 to 0x015247 (0xeba1 bytes) 000143 1767323858.492517 mapread 0x005292 thru 0x00d98e (0x86fd bytes) 000144 1767323858.521636 punch from 0x01488f to 0x019aef (0x5261 bytes) 000145 1767323858.525914 write 0x004229 thru 0x00d579 (0x9351 bytes) 000147 1767323858.528053 trunc from 0x00e7e3 to 0x01183e (0x305c bytes) 000149 1767323858.536895 fallocate 0x023b8a thru 0x02cc12 (0x9089 bytes) 000150 1767323858.541223 read 0x0045fc thru 0x00723c (0x2c41 bytes) 000151 1767323858.541757 trunc from 0x005aed to 0x0092a7 (0x37bb bytes) 000152 1767323858.552469 mapread 0x0229b3 thru 0x02b7c4 (0x8e12 bytes) 000155 1767323858.563265 read 0x0071b1 thru 0x016ece (0xfd1e bytes) 000156 1767323858.564058 mapread 0x0001bd thru 0x00fce4 (0xfb28 bytes) 000157 1767323858.565246 fallocate 0x02a275 thru 0x02cc10 (0x299c bytes) 000158 1767323858.569311 trunc from 0x02de08 to 0x026430 (0x79d7 bytes) 000159 1767323858.586169 write 0x026dba thru 0x03265a (0xb8a1 bytes) 000160 1767323858.587311 mapwrite 0x001354 thru 0x00520d (0x3eba bytes) 000161 1767323858.601099 fallocate 0x02d1a3 thru 0x0368d7 (0x9735 bytes) 000162 1767323858.604931 mapwrite 0x03686d thru 0x03ffff (0x9793 bytes) 000163 1767323858.637203 trunc from 0x040000 to 0x022c0e (0x1d3f1 bytes) 000164 1767323858.649021 mapread 0x01e3d4 thru 0x022c0e (0x483b bytes) 000165 1767323858.650344 write 0x0086cf thru 0x0112a5 (0x8bd7 bytes) 000166 1767323858.651441 write 0x02ff8e thru 0x031d41 (0x1db4 bytes) 000167 1767323858.653512 punch from 0x00ec40 to 0x01e3e9 (0xf7aa bytes) 000168 1767323858.669595 MIRROR_EXTEND to 2 mirrors 000169 1767323859.130787 trunc from 0x014827 to 0x023e94 (0xf66e bytes) 000170 1767323859.157379 fallocate 0x01306a thru 0x017fc7 (0x4f5e bytes) 000171 1767323859.163271 trunc from 0x00b776 to 0x01596c (0xa1f7 bytes) 000172 1767323859.168373 read 0x02ac79 thru 0x02de05 (0x318d bytes) 000173 1767323859.173136 write 0x01ee63 thru 0x020004 (0x11a2 bytes) 000174 1767323859.183653 fallocate 0x02342d thru 0x02f1ef (0xbdc3 bytes) 000175 1767323859.186985 fallocate 0x028686 thru 0x029ae0 (0x145b bytes) 000176 1767323859.189671 mapwrite 0x024bea thru 0x02dbee (0x9005 bytes) 000178 1767323859.208072 trunc from 0x02932c to 0x031d41 (0x8a16 bytes) 000179 1767323859.214666 MIRROR_RESYNC 2 mirrors 000180 1767323859.593913 read 0x017c4f thru 0x01f05a (0x740c bytes) 000181 1767323859.598731 trunc from 0x007785 to 0x00884b (0x10c7 bytes) 000182 1767323859.620696 MIRROR_RESYNC 2 mirrors 000184 1767323860.100830 trunc from 0x0080eb to 0x00f25f (0x7175 bytes) 000185 1767323860.138754 read 0x00c1af thru 0x010a7b (0x48cd bytes) 000186 1767323860.146512 mapwrite 0x00e32a thru 0x013017 (0x4cee bytes) 000187 1767323860.175955 mapread 0x00ef18 thru 0x013652 (0x473b bytes) 000188 1767323860.176846 write 0x02aafb thru 0x02de49 (0x334f bytes) 000189 1767323860.184044 read 0x0255b9 thru 0x031d41 (0xc789 bytes) 000190 1767323860.184677 read 0x01586c thru 0x0200d7 (0xa86c bytes) 000191 1767323860.192924 write 0x005ccc thru 0x00f4f3 (0x9828 bytes) 000192 1767323860.194360 punch from 0x010653 to 0x017e33 (0x77e1 bytes) 000193 1767323860.199927 trunc from 0x031d42 to 0x030fb2 (0xd8f bytes) 000194 1767323860.234073 write 0x03ac0b thru 0x03fa99 (0x4e8f bytes) 000195 1767323860.236022 mapwrite 0x01202b thru 0x021a69 (0xfa3f bytes) 000196 1767323860.276927 fallocate 0x0045cc thru 0x00b226 (0x6c5b bytes) 000197 1767323860.281051 punch from 0x00e0c7 to 0x00fd65 (0x1c9f bytes) 000199 1767323860.285202 fallocate 0x036741 thru 0x03ffff (0x98bf bytes) 000200 1767323860.300789 trunc from 0x0092bc to 0x00c145 (0x2e8a bytes) 000201 1767323860.313174 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x303:0x0]: Invalid argument (22) 000202 1767323860.451880 mapwrite 0x005531 thru 0x00a0dc (0x4bac bytes) 000204 1767323860.496094 trunc from 0x03fa9a to 0x012560 (0x2d539 bytes) 000205 1767323860.519721 mapread 0x005ebe thru 0x00c37a (0x64bd bytes) 000206 1767323860.520467 fallocate 0x03d69e thru 0x03f911 (0x2274 bytes) 000207 1767323860.526013 trunc from 0x03f912 to 0x03eba3 (0xd6e bytes) 000208 1767323860.548334 mapread 0x01045d thru 0x01f87a (0xf41e bytes) 000210 1767323860.558968 trunc from 0x02f112 to 0x03d02e (0xdf1d bytes) 000211 1767323860.562876 fallocate 0x018827 thru 0x023748 (0xaf22 bytes) 000212 1767323860.566292 mapwrite 0x02f9b9 thru 0x0307a8 (0xdf0 bytes) 000213 1767323860.581594 fallocate 0x035ed0 thru 0x03d050 (0x7181 bytes) 000214 1767323860.584890 read 0x01cc36 thru 0x0244d5 (0x78a0 bytes) 000215 1767323860.589904 read 0x014af5 thru 0x01cdc5 (0x82d1 bytes) 000216 1767323860.591160 punch from 0x02c4d0 to 0x031033 (0x4b64 bytes) 000217 1767323860.595726 punch from 0x00f079 to 0x01c283 (0xd20b bytes) 000218 1767323860.599277 punch from 0x0329a4 to 0x036edd (0x453a bytes) 000219 1767323860.601937 read 0x037500 thru 0x03eba3 (0x76a4 bytes) 000220 1767323860.607822 write 0x01a240 thru 0x01b03f (0xe00 bytes) 000222 1767323860.609370 write 0x032ac6 thru 0x03ffff (0xd53a bytes) 000223 1767323860.611055 fallocate 0x02e4a5 thru 0x0375c7 (0x9123 bytes) 000224 1767323860.614475 mapwrite 0x014fe2 thru 0x0196b1 (0x46d0 bytes) 000225 1767323860.628289 punch from 0x020428 to 0x0301b0 (0xfd89 bytes) 000226 1767323860.638418 trunc from 0x040000 to 0x0369f1 (0x960e bytes) 000227 1767323860.662333 punch from 0x016f33 to 0x019511 (0x25df bytes) 000228 1767323860.666896 read 0x00b60a thru 0x00cf22 (0x1919 bytes) 000229 1767323860.667466 trunc from 0x029e15 to 0x035cb0 (0xbe9c bytes) 000231 1767323860.673982 fallocate 0x0186a2 thru 0x019040 (0x99f bytes) 000232 1767323860.678638 write 0x021341 thru 0x02931c (0x7fdc bytes) 000234 1767323860.688480 write 0x03ec89 thru 0x03ffff (0x1377 bytes) 000235 1767323860.692120 trunc from 0x040000 to 0x02417b (0x1be84 bytes) 000236 1767323860.714959 read 0x022eb8 thru 0x023400 (0x549 bytes) 000237 1767323860.718486 MIRROR_EXTEND to 2 mirrors 000238 1767323861.237735 read 0x000222 thru 0x00f03b (0xee1a bytes) 000239 1767323861.245786 trunc from 0x02417c to 0x03b6be (0x17543 bytes) 000240 1767323861.284711 mapread 0x01380e thru 0x020fc1 (0xd7b4 bytes) 000241 1767323861.314280 write 0x004952 thru 0x0132ea (0xe999 bytes) 000242 1767323861.321225 trunc from 0x03b6bf to 0x02ea80 (0xcc3e bytes) 000244 1767323861.335074 write 0x016aa1 thru 0x0200a2 (0x9602 bytes) 000245 1767323861.336322 punch from 0x013a33 to 0x023677 (0xfc45 bytes) 000246 1767323861.355944 mapread 0x00adfd thru 0x016853 (0xba57 bytes) 000248 1767323861.366962 read 0x01be9d thru 0x01bfd2 (0x136 bytes) 000249 1767323861.372490 MIRROR_RESYNC 2 mirrors 000250 1767323861.780756 fallocate 0x039d23 thru 0x03ffff (0x62dd bytes) 000252 1767323861.821668 mapwrite 0x0090a8 thru 0x013862 (0xa7bb bytes) 000253 1767323861.856438 trunc from 0x02ea81 to 0x0295d0 (0x54b0 bytes) 000254 1767323861.894162 mapread 0x023688 thru 0x0295d0 (0x5f49 bytes) 000255 1767323861.906341 trunc from 0x0295d1 to 0x0344d6 (0xaf06 bytes) 000256 1767323861.923983 punch from 0x031ffa to 0x0344d6 (0x24dd bytes) 000257 1767323861.930600 mapwrite 0x028636 thru 0x02abb2 (0x257d bytes) 000258 1767323861.960338 punch from 0x020ce4 to 0x028992 (0x7caf bytes) 000259 1767323861.967780 MIRROR_RESYNC 2 mirrors 000261 1767323862.426341 mapread 0x02ac79 thru 0x02cc0a (0x1f92 bytes) 000262 1767323862.439929 mapwrite 0x024f63 thru 0x034b97 (0xfc35 bytes) 000263 1767323862.517458 read 0x008b89 thru 0x00e9e3 (0x5e5b bytes) 000264 1767323862.527900 mapread 0x0239bb thru 0x025a2b (0x2071 bytes) 000265 1767323862.528736 trunc from 0x011d59 to 0x013139 (0x13e1 bytes) 000266 1767323862.534494 punch from 0x00841c to 0x00ad65 (0x294a bytes) 000267 1767323862.538766 mapwrite 0x03954a thru 0x03ffff (0x6ab6 bytes) 000268 1767323862.568974 trunc from 0x002fbc to 0x010e91 (0xded6 bytes) 000269 1767323862.576518 read 0x00b74d thru 0x011f14 (0x67c8 bytes) 000270 1767323862.583647 mapwrite 0x00f8bb thru 0x0146c0 (0x4e06 bytes) 000271 1767323862.606683 MIRROR_RESYNC 2 mirrors 000273 1767323863.021484 mapwrite 0x02030f thru 0x02f56e (0xf260 bytes) 000275 1767323863.095213 mapread 0x003d98 thru 0x0115d9 (0xd842 bytes) 000276 1767323863.107126 read 0x00578c thru 0x00bdf6 (0x666b bytes) 000277 1767323863.108161 trunc from 0x00bfab to 0x01a0c6 (0xe11c bytes) 000278 1767323863.111874 mapwrite 0x037eab thru 0x03ebfa (0x6d50 bytes) 000279 1767323863.132133 fallocate 0x031fef thru 0x03b7a5 (0x97b7 bytes) 000280 1767323863.135339 punch from 0x032a62 to 0x03d8ca (0xae69 bytes) 000281 1767323863.138784 write 0x032b66 thru 0x03337d (0x818 bytes) 000282 1767323863.143634 trunc from 0x040000 to 0x00d0fb (0x32f04 bytes) 000283 1767323863.163710 fallocate 0x009e90 thru 0x00b2ff (0x1470 bytes) 000284 1767323863.168926 MIRROR_RESYNC 2 mirrors 000285 1767323863.456141 mapwrite 0x03b3e0 thru 0x03ffff (0x4c20 bytes) 000286 1767323863.521837 write 0x029eb3 thru 0x037687 (0xd7d5 bytes) 000287 1767323863.528828 mapread 0x02a647 thru 0x032491 (0x7e4b bytes) 000288 1767323863.529494 mapread 0x037b68 thru 0x03ffff (0x8498 bytes) 000289 1767323863.530272 punch from 0x02d57b to 0x0318bd (0x4343 bytes) 000290 1767323863.539318 fallocate 0x02f51b thru 0x034a26 (0x550c bytes) 000292 1767323863.543858 punch from 0x023bda to 0x027398 (0x37bf bytes) 000293 1767323863.547580 punch from 0x02fece to 0x034776 (0x48a9 bytes) 000294 1767323863.558306 mapwrite 0x00a271 thru 0x011d36 (0x7ac6 bytes) 000295 1767323863.585302 trunc from 0x040000 to 0x021022 (0x1efdd bytes) 000296 1767323863.604352 punch from 0x01f746 to 0x021022 (0x18dd bytes) 000297 1767323863.608707 mapwrite 0x02180c thru 0x02cd8d (0xb582 bytes) 000298 1767323863.647472 trunc from 0x02cd8e to 0x03a1dd (0xd450 bytes) 000299 1767323863.660181 read 0x01eb17 thru 0x02b021 (0xc50b bytes) 000300 1767323863.660682 punch from 0x014543 to 0x01edf6 (0xa8b4 bytes) 000301 1767323863.664238 read 0x01e094 thru 0x02cea8 (0xee15 bytes) 000302 1767323863.673330 punch from 0x035993 to 0x035fd1 (0x63f bytes) 000303 1767323863.678027 trunc from 0x03a1de to 0x02ece6 (0xb4f7 bytes) 000304 1767323863.707217 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x303:0x0]: Invalid argument (22) 000307 1767323863.837684 mapwrite 0x00005e thru 0x00bf12 (0xbeb5 bytes) 000308 1767323863.865378 fallocate 0x020041 thru 0x02bed4 (0xbe94 bytes) 000309 1767323863.870928 trunc from 0x02ece7 to 0x0187ef (0x164f7 bytes) 000310 1767323863.882851 fallocate 0x00a2a9 thru 0x0147bd (0xa515 bytes) 000311 1767323863.886177 mapread 0x0085cd thru 0x010927 (0x835b bytes) 000312 1767323863.886826 punch from 0x000b3f to 0x006ed8 (0x639a bytes) 000313 1767323863.891063 write 0x039fdd thru 0x03d4f8 (0x351c bytes) 000314 1767323863.892347 write 0x03682c thru 0x037bb6 (0x138b bytes) 000315 1767323863.897961 trunc from 0x03d4f9 to 0x03443c (0x90bc bytes) 000316 1767323863.912630 write 0x030bc4 thru 0x03506d (0x44aa bytes) 000317 1767323863.913722 write 0x010818 thru 0x01f719 (0xef02 bytes) 000318 1767323863.920738 read 0x00aa1a thru 0x0191c5 (0xe7ac bytes) 000319 1767323863.921314 mapread 0x02c550 thru 0x03506d (0x8b1e bytes) 000320 1767323863.926659 mapwrite 0x003009 thru 0x01283e (0xf836 bytes) 000321 1767323863.956000 read 0x00f0e7 thru 0x01d5dc (0xe4f6 bytes) 000322 1767323863.956424 mapwrite 0x021644 thru 0x030be1 (0xf59e bytes) 000323 1767323863.978200 mapread 0x022825 thru 0x023255 (0xa31 bytes) 000324 1767323863.978843 trunc from 0x03506e to 0x00c92f (0x2873e bytes) 000325 1767323863.991657 fallocate 0x01b056 thru 0x027c2c (0xcbd7 bytes) 000326 1767323863.994677 read 0x00e2c2 thru 0x01b041 (0xcd80 bytes) 000327 1767323864.004651 mapwrite 0x03963f thru 0x03cd75 (0x3737 bytes) 000328 1767323864.051800 fallocate 0x03f65d thru 0x03ffff (0x9a3 bytes) 000330 1767323864.059150 mapread 0x015f88 thru 0x01a404 (0x447d bytes) 000331 1767323864.060059 fallocate 0x036209 thru 0x03aae1 (0x48d9 bytes) 000332 1767323864.065618 punch from 0x02ccb3 to 0x031548 (0x4896 bytes) 000333 1767323864.074106 trunc from 0x040000 to 0x0238cf (0x1c730 bytes) 000335 1767323864.095257 trunc from 0x0238d0 to 0x03fd5a (0x1c48b bytes) 000336 1767323864.117194 read 0x019a11 thru 0x01a93f (0xf2f bytes) 000337 1767323864.117930 mapwrite 0x0077fe thru 0x00e9f2 (0x71f5 bytes) 000339 1767323864.134677 write 0x004238 thru 0x00b068 (0x6e31 bytes) 000340 1767323864.135815 punch from 0x02345a to 0x0301f9 (0xcda0 bytes) 000341 1767323864.142422 trunc from 0x00eeb6 to 0x01acb7 (0xbe02 bytes) 000342 1767323864.147737 trunc from 0x03fd5b to 0x014063 (0x2bcf7 bytes) 000344 1767323864.164430 mapwrite 0x001a45 thru 0x00eb8b (0xd147 bytes) 000345 1767323864.193143 write 0x0309d7 thru 0x0338d7 (0x2f01 bytes) 000346 1767323864.193924 read 0x019e47 thru 0x01d237 (0x33f1 bytes) 000347 1767323864.198799 trunc from 0x0338d8 to 0x0066ef (0x2d1e8 bytes) 000350 1767323864.219110 trunc from 0x0066f0 to 0x012727 (0xc038 bytes) 000351 1767323864.237451 trunc from 0x012728 to 0x030afe (0x1e3d7 bytes) 000352 1767323864.270902 mapwrite 0x032cd4 thru 0x034d05 (0x2032 bytes) 000353 1767323864.328943 read 0x02dd94 thru 0x034d05 (0x6f72 bytes) 000354 1767323864.339852 mapwrite 0x01b642 thru 0x021d2a (0x66e9 bytes) 000355 1767323864.364896 MIRROR_EXTEND to 2 mirrors 000356 1767323864.745602 read 0x0303b3 thru 0x031b76 (0x17c4 bytes) 000357 1767323864.751061 read 0x010c5c thru 0x01ded2 (0xd277 bytes) 000358 1767323864.766231 mapread 0x01a8c7 thru 0x01e07b (0x37b5 bytes) 000359 1767323864.766953 fallocate 0x03b138 thru 0x03ffff (0x4ec8 bytes) 000360 1767323864.793263 mapread 0x001e75 thru 0x004cdc (0x2e68 bytes) 000362 1767323864.816547 read 0x0168bc thru 0x0198aa (0x2fef bytes) 000363 1767323864.829754 trunc from 0x019f61 to 0x01ae41 (0xee1 bytes) 000364 1767323864.838339 write 0x032e32 thru 0x03ffff (0xd1ce bytes) 000365 1767323864.843964 mapread 0x017dd8 thru 0x01ffbd (0x81e6 bytes) 000366 1767323864.851733 fallocate 0x01823d thru 0x024b75 (0xc939 bytes) 000367 1767323864.858438 trunc from 0x018f01 to 0x028356 (0xf456 bytes) 000368 1767323864.862743 read 0x022f15 thru 0x03226b (0xf357 bytes) 000369 1767323864.875197 fallocate 0x004d56 thru 0x00a3db (0x5686 bytes) 000370 1767323864.885035 punch from 0x03a665 to 0x03ddaa (0x3746 bytes) 000371 1767323864.907296 read 0x032208 thru 0x037b46 (0x593f bytes) 000372 1767323864.908270 fallocate 0x02413d thru 0x025080 (0xf44 bytes) 000373 1767323864.913466 write 0x03c39a thru 0x03ffff (0x3c66 bytes) 000374 1767323864.918145 read 0x0293a1 thru 0x031554 (0x81b4 bytes) 000375 1767323864.924102 MIRROR_RESYNC 2 mirrors 000376 1767323865.404015 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x303:0x0]: Invalid argument (22) 000377 1767323865.510399 punch from 0x001988 to 0x00f8d5 (0xdf4e bytes) 000378 1767323865.521021 mapread 0x00fb63 thru 0x01bab7 (0xbf55 bytes) 000379 1767323865.539867 fallocate 0x0278b3 thru 0x0369b9 (0xf107 bytes) 000380 1767323865.545913 punch from 0x0039c4 to 0x008673 (0x4cb0 bytes) 000381 1767323865.551080 punch from 0x01fa61 to 0x022d69 (0x3309 bytes) 000382 1767323865.555946 mapread 0x016b57 thru 0x01f558 (0x8a02 bytes) 000384 1767323865.568865 MIRROR_EXTEND to 2 mirrors 000385 1767323866.055390 MIRROR_RESYNC 2 mirrors 000386 1767323866.350957 trunc from 0x040000 to 0x01b29f (0x24d60 bytes) 000387 1767323866.402081 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1b2a0): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b2a0): Mirror 2: 0xaf0d8e56 (stale) CRC-32 checksum value for chunk [0, 0x1b2a0): Mirror 1: 0xaf0d8e56 Mirror 3: 0xaf0d8e56 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b2a0: skipped 000388 1767323866.991322 fallocate 0x03a6ae thru 0x03ead2 (0x4425 bytes) 000389 1767323867.024988 mapwrite 0x03f0c4 thru 0x03fdf7 (0xd34 bytes) 000390 1767323867.061066 fallocate 0x0024fe thru 0x0028a1 (0x3a4 bytes) 000392 1767323867.066097 fallocate 0x0101da thru 0x015651 (0x5478 bytes) 000393 1767323867.071220 fallocate 0x03334e thru 0x03fac8 (0xc77b bytes) 000394 1767323867.077178 fallocate 0x008af3 thru 0x00a99f (0x1ead bytes) 000395 1767323867.082333 read 0x0233cd thru 0x026213 (0x2e47 bytes) 000396 1767323867.094774 write 0x02660b thru 0x02ec37 (0x862d bytes) 000397 1767323867.104750 mapread 0x01de6b thru 0x0244de (0x6674 bytes) 000398 1767323867.123161 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x303:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fdf8): Mirror 2: 0xfb9e0c91 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fdf8): Mirror 2: 0xfb9e0c91 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fdf8: skipped 000399 1767323867.331060 read 0x02cbb2 thru 0x03c4b5 (0xf904 bytes) 000400 1767323867.337242 mapread 0x02bc34 thru 0x038b8b (0xcf58 bytes) 000401 1767323867.340991 read 0x019371 thru 0x029136 (0xfdc6 bytes) 000402 1767323867.357216 trunc from 0x03fdf8 to 0x03bf0f (0x3ee8 bytes) 000403 1767323867.374860 MIRROR_RESYNC 2 mirrors 000404 1767323867.778851 mapread 0x0342ca thru 0x03bf0f (0x7c46 bytes) 000406 1767323867.793573 trunc from 0x03bf10 to 0x00fd8a (0x2c185 bytes) 000407 1767323867.832586 read 0x00c73e thru 0x00fd8a (0x364d bytes) 000408 1767323867.837916 read 0x0085cb thru 0x00fd8a (0x77c0 bytes) 000409 1767323867.847237 fallocate 0x034099 thru 0x03de38 (0x9da0 bytes) 000410 1767323867.853415 trunc from 0x00fd8b to 0x01a814 (0xaa8a bytes) 000411 1767323867.872656 mapwrite 0x0363a5 thru 0x039e18 (0x3a74 bytes) 000412 1767323867.926288 fallocate 0x01f1ae thru 0x026dbc (0x7c0f bytes) 000413 1767323867.936772 trunc from 0x022cae to 0x032383 (0xf6d6 bytes) 000415 1767323867.946361 trunc from 0x013378 to 0x013d1a (0x9a3 bytes) 000416 1767323867.951408 trunc from 0x039e19 to 0x038429 (0x19ef bytes) 000417 1767323867.967414 write 0x00acd5 thru 0x0183a8 (0xd6d4 bytes) 000418 1767323867.970159 fallocate 0x002838 thru 0x005192 (0x295b bytes) 000419 1767323867.974143 read 0x02d3b6 thru 0x037ca7 (0xa8f2 bytes) 000420 1767323867.982082 punch from 0x014ea6 to 0x019f48 (0x50a3 bytes) 000421 1767323868.000043 trunc from 0x0326b4 to 0x038429 (0x5d76 bytes) 000422 1767323868.007133 trunc from 0x03842a to 0x0096a4 (0x2ed85 bytes) 000423 1767323868.025108 mapread 0x003071 thru 0x0096a4 (0x6634 bytes) 000424 1767323868.026814 trunc from 0x008aac to 0x00960b (0xb60 bytes) 000425 1767323868.032413 mapread 0x007a82 thru 0x0096a4 (0x1c23 bytes) 000426 1767323868.053277 mapwrite 0x03b4a7 thru 0x03ffff (0x4b59 bytes) 000427 1767323868.090511 punch from 0x023905 to 0x030d16 (0xd412 bytes) 000428 1767323868.094651 mapread 0x00a382 thru 0x00a91c (0x59b bytes) 000430 1767323868.100101 punch from 0x00e087 to 0x017c1e (0x9b98 bytes) 000431 1767323868.103811 trunc from 0x014917 to 0x024130 (0xf81a bytes) 000432 1767323868.107076 trunc from 0x02906c to 0x03114d (0x80e2 bytes) 000433 1767323868.110890 punch from 0x0178c0 to 0x01b7ce (0x3f0f bytes) 000434 1767323868.117231 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x224a84ce (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd77d1d52 Mirror 3: 0xd77d1d52 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000435 1767323868.599969 mapwrite 0x0005f0 thru 0x0089f8 (0x8409 bytes) 000437 1767323868.674712 trunc from 0x001c3e to 0x00ffef (0xe3b2 bytes) 000438 1767323868.679113 punch from 0x0312fb to 0x03e821 (0xd527 bytes) 000439 1767323868.684783 mapwrite 0x0050ec thru 0x01347c (0xe391 bytes) 000441 1767323868.715017 mapread 0x037905 thru 0x03ffff (0x86fb bytes) 000442 1767323868.725982 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x303:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd77d1d52 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd77d1d52 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000443 1767323868.891445 trunc from 0x0280ab to 0x03480b (0xc761 bytes) 000444 1767323868.898087 trunc from 0x040000 to 0x03c0a9 (0x3f56 bytes) 000446 1767323868.934085 write 0x01827d thru 0x02373f (0xb4c3 bytes) 000447 1767323868.942458 write 0x00a8db thru 0x011d1f (0x7445 bytes) 000448 1767323868.952129 punch from 0x035fd8 to 0x039b61 (0x3b8a bytes) 000449 1767323868.957142 trunc from 0x03c0aa to 0x02f6a0 (0xca09 bytes) 000450 1767323868.970661 fallocate 0x010422 thru 0x016454 (0x6033 bytes) 000451 1767323868.977039 write 0x00a40d thru 0x016ce8 (0xc8dc bytes) 000452 1767323868.983464 trunc from 0x02f6a1 to 0x00a735 (0x24f6b bytes) 000453 1767323869.012812 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xa736): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa736): Mirror 3: 0x58d3c50f (stale) CRC-32 checksum value for chunk [0, 0xa736): Mirror 1: 0x908ff737 Mirror 4: 0x908ff737 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa736: skipped 000454 1767323869.607651 trunc from 0x00a736 to 0x02db9c (0x23467 bytes) 000456 1767323869.654518 punch from 0x022318 to 0x0295dd (0x72c6 bytes) 000457 1767323869.657721 read 0x0017d8 thru 0x010162 (0xe98b bytes) 000458 1767323869.663874 mapwrite 0x003cdf thru 0x0054e8 (0x180a bytes) 000459 1767323869.678210 fallocate 0x036b6b thru 0x03c3e1 (0x5877 bytes) 000460 1767323869.688322 mapread 0x02bca0 thru 0x02db9c (0x1efd bytes) 000462 1767323869.708383 write 0x00cccd thru 0x0167e3 (0x9b17 bytes) 000463 1767323869.713916 write 0x0024aa thru 0x00e68a (0xc1e1 bytes) 000465 1767323869.715394 punch from 0x015e7d to 0x0194b3 (0x3637 bytes) 000466 1767323869.728507 mapread 0x00a4fa thru 0x019599 (0xf0a0 bytes) 000467 1767323869.739675 trunc from 0x020bef to 0x02db9c (0xcfae bytes) 000468 1767323869.746986 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2db9d): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2db9d): Mirror 3: 0x32d33182 (stale) Mirror 4: 0xc924761b (stale) CRC-32 checksum value for chunk [0, 0x2db9d): Mirror 1: 0xca60cdef Mirror 5: 0xca60cdef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2db9d: skipped 000469 1767323870.224990 fallocate 0x03fa37 thru 0x03fadd (0xa7 bytes) 000471 1767323870.261913 mapwrite 0x039a5d thru 0x03d763 (0x3d07 bytes) 000472 1767323870.283933 read 0x018f3d thru 0x020e61 (0x7f25 bytes) 000473 1767323870.289822 read 0x03ecb7 thru 0x03fadd (0xe27 bytes) 000474 1767323870.290210 write 0x03c529 thru 0x03ffff (0x3ad7 bytes) 000475 1767323870.294780 read 0x001c3e thru 0x004627 (0x29ea bytes) 000476 1767323870.305237 trunc from 0x00b3e7 to 0x00c243 (0xe5d bytes) 000477 1767323870.309633 fallocate 0x00b260 thru 0x013fda (0x8d7b bytes) 000478 1767323870.315078 punch from 0x039b45 to 0x03ffff (0x64bb bytes) 000479 1767323870.334393 MIRROR_RESYNC 4 mirrors 000481 1767323870.942521 punch from 0x02eb6b to 0x033f55 (0x53eb bytes) 000482 1767323870.968405 fallocate 0x03fea3 thru 0x03ffff (0x15d bytes) 000483 1767323870.971923 write 0x00425d thru 0x00c4cf (0x8273 bytes) 000484 1767323870.976911 write 0x0325f2 thru 0x03d0a5 (0xaab4 bytes) 000485 1767323870.987227 mapwrite 0x03d290 thru 0x03ffff (0x2d70 bytes) 000486 1767323871.004230 trunc from 0x040000 to 0x00ee13 (0x311ec bytes) 000488 1767323871.035259 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x303:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xee14): Mirror 4: 0x78b8063 (stale) Mirror 5: 0x78b8063 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xee14): Mirror 4: 0x78b8063 (stale) Mirror 5: 0x78b8063 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xee14: skipped 000489 1767323871.232028 mapread 0x00bdd4 thru 0x00ee13 (0x3040 bytes) 000490 1767323871.242091 write 0x03cb10 thru 0x03ffff (0x34f0 bytes) 000491 1767323871.249099 mapread 0x02c84a thru 0x03aa74 (0xe22b bytes) 000492 1767323871.257726 read 0x0136ab thru 0x014edb (0x1831 bytes) 000493 1767323871.261068 trunc from 0x040000 to 0x00c446 (0x33bb9 bytes) 000494 1767323871.274408 MIRROR_RESYNC 3 mirrors 000495 1767323871.622007 MIRROR_RESYNC 3 mirrors 000496 1767323871.954657 mapread 0x00972f thru 0x00c446 (0x2d18 bytes) 000497 1767323871.983178 read 0x00c379 thru 0x00c446 (0xce bytes) 000498 1767323871.987680 MIRROR_EXTEND to 4 mirrors 000500 1767323872.485212 trunc from 0x00b125 to 0x00c446 (0x1322 bytes) 000501 1767323872.538839 punch from 0x00152d to 0x00c446 (0xaf1a bytes) 000502 1767323872.547685 mapread 0x009225 thru 0x00c446 (0x3222 bytes) 000503 1767323872.565582 punch from 0x002ab1 to 0x00c446 (0x9996 bytes) 000505 1767323872.575912 punch from 0x005f20 to 0x00c446 (0x6527 bytes) 000506 1767323872.585313 fallocate 0x03e3e5 thru 0x03ffff (0x1c1b bytes) 000507 1767323872.589933 read 0x03e227 thru 0x03ffff (0x1dd9 bytes) 000508 1767323872.595182 fallocate 0x035eab thru 0x0383d2 (0x2528 bytes) 000510 1767323872.603261 mapwrite 0x0123ee thru 0x017cb1 (0x58c4 bytes) 000511 1767323872.636930 punch from 0x03e0db to 0x03ffff (0x1f25 bytes) 000514 1767323872.646461 mapread 0x013095 thru 0x021eff (0xee6b bytes) 000515 1767323872.657470 trunc from 0x040000 to 0x01df58 (0x220a7 bytes) 000517 1767323872.686831 fallocate 0x03f5c0 thru 0x03ffff (0xa40 bytes) 000518 1767323872.693878 mapread 0x015da4 thru 0x017c77 (0x1ed4 bytes) 000519 1767323872.695643 mapwrite 0x003ef5 thru 0x00bccb (0x7dd7 bytes) 000520 1767323872.725272 mapread 0x01c3c7 thru 0x01df58 (0x1b92 bytes) 000521 1767323872.726415 trunc from 0x0167b4 to 0x01df58 (0x77a5 bytes) 000524 1767323872.743086 punch from 0x011642 to 0x013d12 (0x26d1 bytes) 000525 1767323872.755149 write 0x002b34 thru 0x011c2e (0xf0fb bytes) 000526 1767323872.761097 write 0x015901 thru 0x0239f3 (0xe0f3 bytes) 000527 1767323872.763056 mapwrite 0x033b87 thru 0x034dc4 (0x123e bytes) 000528 1767323872.812168 mapread 0x01b9b8 thru 0x02975c (0xdda5 bytes) 000529 1767323872.818793 read 0x009099 thru 0x016a83 (0xd9eb bytes) 000530 1767323872.825829 punch from 0x0050ae to 0x011a35 (0xc988 bytes) 000531 1767323872.832176 trunc from 0x002448 to 0x0039d9 (0x1592 bytes) 000532 1767323872.836363 trunc from 0x034dc5 to 0x000166 (0x34c5e bytes) 000533 1767323872.855923 trunc from 0x000167 to 0x0169ac (0x16846 bytes) 000534 1767323872.867978 mapwrite 0x013df9 thru 0x021452 (0xd65a bytes) 000535 1767323872.987176 MIRROR_RESYNC 4 mirrors 000536 1767323873.590439 MIRROR_EXTEND to 5 mirrors 000537 1767323874.067478 trunc from 0x010170 to 0x0148bb (0x474c bytes) 000539 1767323874.098200 fallocate 0x02372c thru 0x0276a0 (0x3f75 bytes) 000540 1767323874.101865 trunc from 0x0079ff to 0x01331c (0xb91e bytes) 000541 1767323874.109084 punch from 0x00144a to 0x005ecd (0x4a84 bytes) 000542 1767323874.112776 punch from 0x009515 to 0x018963 (0xf44f bytes) 000543 1767323874.121758 read 0x019fec thru 0x021452 (0x7467 bytes) 000544 1767323874.139861 fallocate 0x00bb50 thru 0x01a467 (0xe918 bytes) 000545 1767323874.143884 trunc from 0x021453 to 0x037b1f (0x166cd bytes) 000546 1767323874.160776 fallocate 0x00b643 thru 0x00e99f (0x335d bytes) 000547 1767323874.163799 punch from 0x001102 to 0x005958 (0x4857 bytes) 000548 1767323874.166462 trunc from 0x02a3ee to 0x0323a1 (0x7fb4 bytes) 000549 1767323874.173175 punch from 0x019ed1 to 0x0218a6 (0x79d6 bytes) 000550 1767323874.177885 read 0x035999 thru 0x03770f (0x1d77 bytes) 000552 1767323874.186988 fallocate 0x01d334 thru 0x026327 (0x8ff4 bytes) 000553 1767323874.190642 trunc from 0x00606f to 0x006dab (0xd3d bytes) 000554 1767323874.196075 read 0x00a9c4 thru 0x0199f3 (0xf030 bytes) 000555 1767323874.197172 trunc from 0x02ff23 to 0x033a31 (0x3b0f bytes) 000556 1767323874.200855 trunc from 0x00f8b7 to 0x014864 (0x4fae bytes) 000557 1767323874.203792 read 0x020177 thru 0x02d292 (0xd11c bytes) 000558 1767323874.204230 trunc from 0x037b20 to 0x010d67 (0x26db8 bytes) 000559 1767323874.213573 mapread 0x00aa1f thru 0x00e9d3 (0x3fb5 bytes) 000560 1767323874.214362 punch from 0x00a733 to 0x010d67 (0x6635 bytes) 000561 1767323874.218122 fallocate 0x01d3a6 thru 0x028fdc (0xbc37 bytes) 000562 1767323874.223366 read 0x000335 thru 0x00b923 (0xb5ef bytes) 000563 1767323874.234876 mapread 0x001802 thru 0x007073 (0x5872 bytes) 000564 1767323874.236461 mapread 0x00625d thru 0x00797c (0x1720 bytes) 000565 1767323874.241454 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x10d68): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10d68): Mirror 4: 0x70f1ff5a (stale) Mirror 5: 0x70f1ff5a (stale) Mirror 6: 0x70f1ff5a (stale) Mirror 7: 0x70f1ff5a (stale) CRC-32 checksum value for chunk [0, 0x10d68): Mirror 1: 0x70f1ff5a Mirror 8: 0x70f1ff5a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10d68: skipped 000566 1767323874.774077 fallocate 0x02ef10 thru 0x03c9a5 (0xda96 bytes) 000567 1767323874.811522 mapwrite 0x03bad7 thru 0x03ffff (0x4529 bytes) 000568 1767323874.879213 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x303:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x91f2eda5 (stale) Mirror 6: 0x91f2eda5 (stale) Mirror 7: 0x91f2eda5 (stale) Mirror 8: 0xd0b24aeb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x91f2eda5 (stale) Mirror 6: 0x91f2eda5 (stale) Mirror 7: 0x91f2eda5 (stale) Mirror 8: 0xd0b24aeb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000569 1767323875.158242 read 0x0006e8 thru 0x0028de (0x21f7 bytes) 000570 1767323875.166602 fallocate 0x033b30 thru 0x03a8dc (0x6dad bytes) 000571 1767323875.174205 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x303:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x91f2eda5 (stale) Mirror 6: 0x91f2eda5 (stale) Mirror 8: 0xd0b24aeb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x91f2eda5 (stale) Mirror 6: 0x91f2eda5 (stale) Mirror 8: 0xd0b24aeb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000572 1767323875.518365 trunc from 0x040000 to 0x0383ce (0x7c31 bytes) 000573 1767323875.534952 mapread 0x030614 thru 0x0366e6 (0x60d3 bytes) 000574 1767323875.548888 fallocate 0x03c185 thru 0x03ffff (0x3e7b bytes) 000575 1767323875.552503 mapwrite 0x0180fc thru 0x02113d (0x9042 bytes) 000576 1767323875.580345 mapwrite 0x037267 thru 0x03a355 (0x30ef bytes) 000578 1767323875.613141 mapread 0x02f4f0 thru 0x03369c (0x41ad bytes) 000579 1767323875.614022 read 0x009b18 thru 0x018260 (0xe749 bytes) 000580 1767323875.622925 trunc from 0x0315cd to 0x0321b4 (0xbe8 bytes) 000581 1767323875.628584 mapread 0x00e064 thru 0x018aa1 (0xaa3e bytes) 000582 1767323875.629656 trunc from 0x01489a to 0x0221b3 (0xd91a bytes) 000583 1767323875.634501 trunc from 0x040000 to 0x02a739 (0x158c6 bytes) 000584 1767323875.651606 mapwrite 0x033b91 thru 0x039315 (0x5785 bytes) 000585 1767323875.680860 mapwrite 0x031fa6 thru 0x03aecd (0x8f28 bytes) 000586 1767323875.713833 read 0x023ae9 thru 0x02dcea (0xa202 bytes) 000587 1767323875.723919 read 0x005fdd thru 0x0149eb (0xea0f bytes) 000588 1767323875.725071 trunc from 0x03aece to 0x005512 (0x359bb bytes) 000589 1767323875.737786 mapwrite 0x00770b thru 0x0171b2 (0xfaa8 bytes) 000590 1767323875.771685 mapread 0x01123f thru 0x0131b2 (0x1f74 bytes) 000591 1767323875.772319 trunc from 0x00c251 to 0x01697a (0xa72a bytes) 000592 1767323875.775782 fallocate 0x01c94e thru 0x02b735 (0xede8 bytes) 000593 1767323875.781479 read 0x002b78 thru 0x00831a (0x57a3 bytes) 000594 1767323875.782577 write 0x01a407 thru 0x022e7d (0x8a77 bytes) 000595 1767323875.783923 trunc from 0x01f31a to 0x022e7d (0x3b64 bytes) 000596 1767323875.796833 MIRROR_RESYNC 4 mirrors 000597 1767323876.553658 mapread 0x011822 thru 0x01ecf3 (0xd4d2 bytes) 000598 1767323876.573912 fallocate 0x03bda7 thru 0x03ffff (0x4259 bytes) 000599 1767323876.598859 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x22e7e): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22e7e): Mirror 5: 0xcb64eb15 (stale) Mirror 6: 0xcb64eb15 (stale) Mirror 8: 0xcb64eb15 (stale) CRC-32 checksum value for chunk [0, 0x22e7e): Mirror 1: 0xcb64eb15 Mirror 9: 0xcb64eb15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22e7e: skipped 000600 1767323877.202719 trunc from 0x021a67 to 0x022e7d (0x1417 bytes) 000601 1767323877.223845 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x22e7e): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22e7e): Mirror 5: 0xcb64eb15 (stale) Mirror 6: 0xcb64eb15 (stale) Mirror 8: 0xcb64eb15 (stale) Mirror 9: 0xcb64eb15 (stale) CRC-32 checksum value for chunk [0, 0x22e7e): Mirror 1: 0xcb64eb15 Mirror 10: 0xcb64eb15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22e7e: skipped 000602 1767323877.862665 trunc from 0x022e7e to 0x03d06c (0x1a1ef bytes) 000603 1767323877.898365 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x303:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d06d): Mirror 5: 0x42c67154 (stale) Mirror 6: 0x42c67154 (stale) Mirror 8: 0x42c67154 (stale) Mirror 10: 0x42c67154 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d06d): Mirror 5: 0x42c67154 (stale) Mirror 6: 0x42c67154 (stale) Mirror 8: 0x42c67154 (stale) Mirror 10: 0x42c67154 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d06d: skipped 000604 1767323878.203928 trunc from 0x03d06d to 0x01addd (0x2228f bytes) 000605 1767323878.214635 read 0x005315 thru 0x00fa1f (0xa70b bytes) 000606 1767323878.219660 punch from 0x006421 to 0x00c32b (0x5f0b bytes) 000607 1767323878.223105 mapwrite 0x037c3b thru 0x03f02a (0x73f0 bytes) 000608 1767323878.260385 punch from 0x01669e to 0x018945 (0x22a8 bytes) 000609 1767323878.268205 write 0x039f52 thru 0x03b490 (0x153f bytes) 000610 1767323878.272686 MIRROR_RESYNC 5 mirrors 000613 1767323879.424048 punch from 0x02f82f to 0x032713 (0x2ee5 bytes) 000614 1767323879.439348 mapwrite 0x037aa2 thru 0x03db03 (0x6062 bytes) 000615 1767323879.456906 punch from 0x03870a to 0x03f02a (0x6921 bytes) 000616 1767323879.459999 mapread 0x022a12 thru 0x026b89 (0x4178 bytes) 000618 1767323879.466032 fallocate 0x021a60 thru 0x02e30a (0xc8ab bytes) 000619 1767323879.468564 read 0x0245b9 thru 0x02ab1e (0x6566 bytes) 000620 1767323879.471150 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3f02b): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f02b): Mirror 5: 0x8df36eae (stale) Mirror 6: 0x8df36eae (stale) Mirror 8: 0x8df36eae (stale) Mirror 10: 0x8df36eae (stale) CRC-32 checksum value for chunk [0, 0x3f02b): Mirror 1: 0x624ae059 Mirror 11: 0x624ae059 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f02b: skipped 000621 1767323880.009172 mapread 0x0117e6 thru 0x0144dd (0x2cf8 bytes) 000622 1767323880.023758 mapread 0x02989a thru 0x0323fe (0x8b65 bytes) 000623 1767323880.034245 punch from 0x02b1ea to 0x03877c (0xd593 bytes) 000624 1767323880.060040 punch from 0x02085e to 0x025e97 (0x563a bytes) 000625 1767323880.062999 mapwrite 0x00ab91 thru 0x01a244 (0xf6b4 bytes) 000626 1767323880.083115 fallocate 0x00dca0 thru 0x01178e (0x3aef bytes) 000627 1767323880.085776 mapwrite 0x0388d7 thru 0x03e2a8 (0x59d2 bytes) 000629 1767323880.100854 MIRROR_RESYNC 6 mirrors 000630 1767323880.902790 trunc from 0x03f02b to 0x02fab4 (0xf576 bytes) 000631 1767323880.938458 mapread 0x00948b thru 0x0190d9 (0xfc4f bytes) 000632 1767323880.951840 punch from 0x00cab8 to 0x01bed2 (0xf41b bytes) 000633 1767323880.956016 trunc from 0x02fab5 to 0x0000ef (0x2f9c5 bytes) 000634 1767323880.968096 punch from 0x0000c5 to 0x0000ef (0x2b bytes) 000635 1767323880.972043 trunc from 0x000090 to 0x0000ef (0x60 bytes) 000636 1767323880.981149 mapread 0x000043 thru 0x0000ef (0xad bytes) 000637 1767323880.985308 trunc from 0x0000e6 to 0x0000ef (0xa bytes) 000639 1767323880.992295 mapwrite 0x03f016 thru 0x03ffff (0xfea bytes) 000640 1767323881.036392 trunc from 0x01809b to 0x02328c (0xb1f2 bytes) 000641 1767323881.039968 punch from 0x00f671 to 0x010ad0 (0x1460 bytes) 000642 1767323881.043646 mapread 0x01f849 thru 0x02ab5c (0xb314 bytes) 000643 1767323881.056730 read 0x02e656 thru 0x0303ac (0x1d57 bytes) 000645 1767323881.057972 punch from 0x01f31f to 0x02e409 (0xf0eb bytes) 000646 1767323881.061210 read 0x0387ad thru 0x03ffff (0x7853 bytes) 000647 1767323881.061787 write 0x02923c thru 0x02fff9 (0x6dbe bytes) 000648 1767323881.070066 mapread 0x014b3d thru 0x023035 (0xe4f9 bytes) 000649 1767323881.070864 trunc from 0x02d95f to 0x036622 (0x8cc4 bytes) 000650 1767323881.080458 mapwrite 0x02876e thru 0x02b54f (0x2de2 bytes) 000651 1767323881.090985 mapread 0x012970 thru 0x0166fa (0x3d8b bytes) 000652 1767323881.091980 mapwrite 0x0231e7 thru 0x025d5a (0x2b74 bytes) 000653 1767323881.104358 punch from 0x01de5c to 0x0268e0 (0x8a85 bytes) 000654 1767323881.108037 mapwrite 0x00373f thru 0x00fe70 (0xc732 bytes) 000655 1767323881.121481 fallocate 0x018874 thru 0x0249f8 (0xc185 bytes) 000656 1767323881.125573 read 0x007e36 thru 0x0090bf (0x128a bytes) 000657 1767323881.126491 mapread 0x011287 thru 0x01c0d1 (0xae4b bytes) 000658 1767323881.127462 fallocate 0x01b086 thru 0x020402 (0x537d bytes) 000660 1767323881.132294 read 0x0348a4 thru 0x03ffff (0xb75c bytes) 000661 1767323881.142070 trunc from 0x00f87a to 0x01e830 (0xefb7 bytes) 000662 1767323881.147506 mapwrite 0x02d550 thru 0x03d03f (0xfaf0 bytes) 000663 1767323881.159799 trunc from 0x01710e to 0x01aa46 (0x3939 bytes) 000664 1767323881.162712 punch from 0x03abb4 to 0x03cced (0x213a bytes) 000665 1767323881.166479 trunc from 0x040000 to 0x01467c (0x2b983 bytes) 000666 1767323881.180332 trunc from 0x01467d to 0x01565f (0xfe3 bytes) 000667 1767323881.189885 write 0x00faef thru 0x011ef6 (0x2408 bytes) 000668 1767323881.193788 trunc from 0x00138d to 0x00d133 (0xbda7 bytes) 000669 1767323881.200414 fallocate 0x00d999 thru 0x017ce5 (0xa34d bytes) 000670 1767323881.204204 fallocate 0x01bfee thru 0x0237fb (0x780e bytes) 000671 1767323881.206885 trunc from 0x0116c5 to 0x017ce5 (0x6621 bytes) 000672 1767323881.214191 punch from 0x0097a9 to 0x017ce5 (0xe53d bytes) 000674 1767323881.220593 fallocate 0x01f5ca thru 0x02e72c (0xf163 bytes) 000675 1767323881.234875 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x303:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17ce6): Mirror 5: 0x5bc646f5 (stale) Mirror 6: 0x5bc646f5 (stale) Mirror 8: 0x5bc646f5 (stale) Mirror 11: 0x5bc646f5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17ce6): Mirror 5: 0x5bc646f5 (stale) Mirror 6: 0x5bc646f5 (stale) Mirror 8: 0x5bc646f5 (stale) Mirror 11: 0x5bc646f5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17ce6: skipped 000676 1767323881.531843 mapread 0x005da2 thru 0x0159d0 (0xfc2f bytes) 000677 1767323881.557064 punch from 0x00e5b8 to 0x013250 (0x4c99 bytes) 000678 1767323881.560758 write 0x0063b9 thru 0x014bbf (0xe807 bytes) 000679 1767323881.564028 write 0x02a69d thru 0x032a9d (0x8401 bytes) 000680 1767323881.571269 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x303:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32a9e): Mirror 5: 0xc3f190c5 (stale) Mirror 6: 0xc3f190c5 (stale) Mirror 8: 0xc3f190c5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32a9e): Mirror 5: 0xc3f190c5 (stale) Mirror 6: 0xc3f190c5 (stale) Mirror 8: 0xc3f190c5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32a9e: skipped 000681 1767323881.820719 write 0x0197c8 thru 0x01ac7e (0x14b7 bytes) 000682 1767323881.829919 trunc from 0x032a9e to 0x00f4ba (0x235e3 bytes) 000684 1767323881.845017 mapwrite 0x026f61 thru 0x03419f (0xd23f bytes) 000685 1767323881.877474 punch from 0x011255 to 0x015825 (0x45d1 bytes) 000686 1767323881.880716 fallocate 0x014145 thru 0x0238e0 (0xf79c bytes) 000688 1767323881.884057 punch from 0x03300a to 0x03419f (0x1196 bytes) 000690 1767323881.887793 trunc from 0x024745 to 0x0307e0 (0xc09c bytes) 000691 1767323881.891761 mapread 0x01cc0b thru 0x02aea6 (0xe29c bytes) 000692 1767323881.903288 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x303:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x341a0): Mirror 5: 0xcaa86fd5 (stale) Mirror 8: 0xcaa86fd5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x341a0): Mirror 5: 0xcaa86fd5 (stale) Mirror 8: 0xcaa86fd5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x341a0: skipped 000693 1767323882.098069 fallocate 0x02a847 thru 0x02e0cc (0x3886 bytes) 000694 1767323882.106803 write 0x00c3fd thru 0x01972a (0xd32e bytes) 000695 1767323882.120832 MIRROR_RESYNC 3 mirrors 000696 1767323882.702574 MIRROR_EXTEND to 4 mirrors 000697 1767323883.218906 MIRROR_RESYNC 4 mirrors 000698 1767323883.510391 write 0x02c49f thru 0x02fedf (0x3a41 bytes) 000699 1767323883.552071 trunc from 0x00e1d1 to 0x0183af (0xa1df bytes) 000701 1767323883.557100 mapread 0x00e20c thru 0x01b23e (0xd033 bytes) 000702 1767323883.577203 trunc from 0x0260fd to 0x029a70 (0x3974 bytes) 000703 1767323883.585426 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x303:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x341a0): Mirror 5: 0xbfa8d78f (stale) Mirror 9: 0xbfa8d78f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x341a0): Mirror 5: 0xbfa8d78f (stale) Mirror 9: 0xbfa8d78f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x341a0: skipped 000704 1767323883.870869 fallocate 0x00bd9a thru 0x015ce9 (0x9f50 bytes) 000705 1767323883.879507 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x341a0): Mirror 5: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x341a0): Mirror 5: 0xbfa8d78f (stale) Mirror 9: 0xbfa8d78f (stale) CRC-32 checksum value for chunk [0, 0x341a0): Mirror 1: 0x450c0617 Mirror 10: 0x450c0617 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x341a0: skipped 000706 1767323884.545699 mapwrite 0x014546 thru 0x017d61 (0x381c bytes) 000707 1767323884.638516 fallocate 0x01d114 thru 0x021233 (0x4120 bytes) 000708 1767323884.642684 read 0x02e19a thru 0x03419f (0x6006 bytes) 000709 1767323884.648748 write 0x03b5d8 thru 0x03dc40 (0x2669 bytes) 000710 1767323884.652647 mapread 0x01e718 thru 0x02c9d6 (0xe2bf bytes) 000711 1767323884.664027 trunc from 0x036bee to 0x03dc40 (0x7053 bytes) 000712 1767323884.676214 punch from 0x032223 to 0x033bd3 (0x19b1 bytes) 000715 1767323884.683820 fallocate 0x00100a thru 0x010191 (0xf188 bytes) 000716 1767323884.692914 MIRROR_RESYNC 4 mirrors 000717 1767323885.633449 mapwrite 0x009c9d thru 0x00bab0 (0x1e14 bytes) 000718 1767323885.724494 punch from 0x020cea to 0x02c83a (0xbb51 bytes) 000719 1767323885.744051 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3dc41): Mirror 5: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc41): Mirror 5: 0x9a2a4d7c (stale) Mirror 9: 0x9a2a4d7c (stale) Mirror 10: 0x9a2a4d7c (stale) CRC-32 checksum value for chunk [0, 0x3dc41): Mirror 1: 0x48dcf083 Mirror 11: 0x48dcf083 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc41: skipped 000720 1767323886.318896 punch from 0x038ed0 to 0x03dc40 (0x4d71 bytes) 000721 1767323886.354809 punch from 0x015e9d to 0x021f97 (0xc0fb bytes) 000723 1767323886.358638 punch from 0x007347 to 0x00d57e (0x6238 bytes) 000725 1767323886.362874 fallocate 0x02536a thru 0x028366 (0x2ffd bytes) 000726 1767323886.365606 write 0x0364ea thru 0x03ffff (0x9b16 bytes) 000727 1767323886.373118 read 0x0106c3 thru 0x01e9af (0xe2ed bytes) 000728 1767323886.385438 mapwrite 0x015372 thru 0x023ad3 (0xe762 bytes) 000729 1767323886.421917 write 0x01efcd thru 0x0245ca (0x55fe bytes) 000730 1767323886.423498 mapwrite 0x003bf4 thru 0x00e045 (0xa452 bytes) 000731 1767323886.461090 punch from 0x016e60 to 0x022f79 (0xc11a bytes) 000732 1767323886.466443 trunc from 0x040000 to 0x01346b (0x2cb94 bytes) 000733 1767323886.495439 punch from 0x00dd20 to 0x01346b (0x574c bytes) 000734 1767323886.499432 write 0x025e86 thru 0x0278d7 (0x1a52 bytes) 000735 1767323886.500843 fallocate 0x002fd1 thru 0x010042 (0xd072 bytes) 000736 1767323886.512913 MIRROR_RESYNC 5 mirrors 000737 1767323887.251922 fallocate 0x010881 thru 0x01d839 (0xcfb9 bytes) 000738 1767323887.275572 mapread 0x02465c thru 0x0278d7 (0x327c bytes) 000739 1767323887.286797 write 0x03d19c thru 0x03ffff (0x2e64 bytes) 000740 1767323887.287719 fallocate 0x02a082 thru 0x038f68 (0xeee7 bytes) 000741 1767323887.292556 mapwrite 0x0176d4 thru 0x024d5f (0xd68c bytes) 000742 1767323887.317422 fallocate 0x003918 thru 0x00b16f (0x7858 bytes) 000743 1767323887.320031 read 0x01ce94 thru 0x0299c1 (0xcb2e bytes) 000744 1767323887.324368 trunc from 0x03d2cd to 0x03ffff (0x2d33 bytes) 000745 1767323887.328761 fallocate 0x03987a thru 0x03ffff (0x6786 bytes) 000746 1767323887.333327 write 0x03e96f thru 0x03ffff (0x1691 bytes) 000747 1767323887.348807 punch from 0x00a4c9 to 0x0122b0 (0x7de8 bytes) 000748 1767323887.351899 trunc from 0x039065 to 0x03f106 (0x60a2 bytes) 000749 1767323887.361495 mapread 0x030f22 thru 0x0342ae (0x338d bytes) 000750 1767323887.365556 fallocate 0x034982 thru 0x03c493 (0x7b12 bytes) 000751 1767323887.369061 write 0x01d982 thru 0x020e29 (0x34a8 bytes) 000752 1767323887.369881 punch from 0x00068a to 0x0041cd (0x3b44 bytes) 000753 1767323887.372394 read 0x036501 thru 0x03816f (0x1c6f bytes) 000754 1767323887.372884 mapwrite 0x02d622 thru 0x02fa5e (0x243d bytes) 000755 1767323887.384063 read 0x001fe4 thru 0x0101d4 (0xe1f1 bytes) 000756 1767323887.392119 fallocate 0x0150ee thru 0x0160f6 (0x1009 bytes) 000757 1767323887.399733 punch from 0x01efc0 to 0x028b91 (0x9bd2 bytes) 000758 1767323887.404400 trunc from 0x040000 to 0x0228f9 (0x1d706 bytes) 000759 1767323887.415364 mapread 0x0161f7 thru 0x0163dd (0x1e7 bytes) 000761 1767323887.417164 mapwrite 0x01c13c thru 0x0227dc (0x66a1 bytes) 000762 1767323887.455758 trunc from 0x0228fa to 0x00b211 (0x176e8 bytes) 000763 1767323887.469716 mapread 0x0078cc thru 0x00b211 (0x3946 bytes) 000764 1767323887.474473 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x303:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb212): Mirror 9: 0x12bcc572 (stale) Mirror 10: 0x12bcc572 (stale) Mirror 11: 0x12bcc572 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb212): Mirror 9: 0x12bcc572 (stale) Mirror 10: 0x12bcc572 (stale) Mirror 11: 0x12bcc572 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb212: skipped 000765 1767323887.695357 mapwrite 0x019544 thru 0x021f04 (0x89c1 bytes) 000766 1767323887.728358 mapwrite 0x0314d0 thru 0x03d40e (0xbf3f bytes) 000768 1767323887.770065 write 0x03eda4 thru 0x03ffff (0x125c bytes) 000769 1767323887.770855 mapread 0x00ebd8 thru 0x015708 (0x6b31 bytes) 000770 1767323887.776150 mapwrite 0x00f9c8 thru 0x018737 (0x8d70 bytes) 000771 1767323887.788800 trunc from 0x00187d to 0x010850 (0xefd4 bytes) 000772 1767323887.792039 trunc from 0x040000 to 0x01eee2 (0x2111d bytes) 000773 1767323887.810161 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1eee3): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1eee3): Mirror 9: 0xc1dea40e (stale) Mirror 10: 0xc1dea40e (stale) Mirror 11: 0xc1dea40e (stale) CRC-32 checksum value for chunk [0, 0x1eee3): Mirror 1: 0xd9aafe9c Mirror 12: 0xd9aafe9c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1eee3: skipped 000774 1767323888.228144 mapread 0x00fece thru 0x01eee2 (0xf015 bytes) 000775 1767323888.253591 write 0x038a16 thru 0x03ac24 (0x220f bytes) 000776 1767323888.284911 fallocate 0x006d6f thru 0x009f9e (0x3230 bytes) 000777 1767323888.289604 mapwrite 0x031355 thru 0x03ffff (0xecab bytes) 000778 1767323888.366208 mapwrite 0x030909 thru 0x035a70 (0x5168 bytes) 000779 1767323888.380384 mapwrite 0x03abf7 thru 0x03fda0 (0x51aa bytes) 000780 1767323888.397187 mapread 0x023913 thru 0x02b975 (0x8063 bytes) 000781 1767323888.408361 mapread 0x0204f9 thru 0x025c5e (0x5766 bytes) 000782 1767323888.409117 read 0x03711a thru 0x03ffff (0x8ee6 bytes) 000784 1767323888.410155 trunc from 0x0150c5 to 0x01ab37 (0x5a73 bytes) 000785 1767323888.421679 MIRROR_RESYNC 5 mirrors 000786 1767323889.272720 punch from 0x02db92 to 0x03b57e (0xd9ed bytes) 000787 1767323889.298375 mapwrite 0x02e577 thru 0x02e81f (0x2a9 bytes) 000789 1767323889.319729 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x303:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x36979826 (stale) Mirror 11: 0x36979826 (stale) Mirror 12: 0x36979826 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x36979826 (stale) Mirror 11: 0x36979826 (stale) Mirror 12: 0x36979826 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000791 1767323889.550322 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x303:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x36979826 (stale) Mirror 12: 0x36979826 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x36979826 (stale) Mirror 12: 0x36979826 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000792 1767323889.736441 punch from 0x038834 to 0x03bd3b (0x3508 bytes) 000793 1767323889.740063 fallocate 0x01c656 thru 0x01eaa8 (0x2453 bytes) 000794 1767323889.743427 read 0x01ef13 thru 0x01fdca (0xeb8 bytes) 000795 1767323889.746879 write 0x030659 thru 0x0377bd (0x7165 bytes) 000796 1767323889.755612 read 0x006650 thru 0x0086d6 (0x2087 bytes) 000797 1767323889.761584 punch from 0x02f83f to 0x03e387 (0xeb49 bytes) 000798 1767323889.768093 fallocate 0x02353c thru 0x028dd9 (0x589e bytes) 000799 1767323889.770874 mapwrite 0x034d33 thru 0x037731 (0x29ff bytes) 000800 1767323889.796908 trunc from 0x040000 to 0x03a0a9 (0x5f56 bytes) 000801 1767323889.813173 trunc from 0x03a0aa to 0x02b8a3 (0xe806 bytes) 000802 1767323889.827159 mapwrite 0x01b291 thru 0x027ca1 (0xca11 bytes) 000803 1767323889.849710 punch from 0x00bd2d to 0x0195a2 (0xd876 bytes) 000804 1767323889.862233 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2b8a4): Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b8a4): Mirror 11: 0x59c48425 (stale) Mirror 12: 0x59c48425 (stale) CRC-32 checksum value for chunk [0, 0x2b8a4): Mirror 1: 0x11f740a4 Mirror 13: 0x11f740a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b8a4: skipped 000806 1767323890.489089 trunc from 0x02b8a4 to 0x01850f (0x13394 bytes) 000807 1767323890.532657 fallocate 0x0337ad thru 0x03aa3c (0x7290 bytes) 000808 1767323890.536268 mapread 0x01002d thru 0x010b21 (0xaf5 bytes) 000809 1767323890.539296 fallocate 0x02c2aa thru 0x034212 (0x7f69 bytes) 000810 1767323890.542805 write 0x01c791 thru 0x02c509 (0xfd79 bytes) 000811 1767323890.549193 trunc from 0x03aa3d to 0x025519 (0x15523 bytes) 000812 1767323890.564818 read 0x018992 thru 0x022650 (0x9cbf bytes) 000814 1767323890.570678 mapwrite 0x03e08f thru 0x03ffff (0x1f71 bytes) 000815 1767323890.610208 read 0x00228e thru 0x002a1f (0x792 bytes) 000816 1767323890.619615 punch from 0x02af89 to 0x03367d (0x86f5 bytes) 000817 1767323890.626606 punch from 0x00348d to 0x00de77 (0xa9eb bytes) 000818 1767323890.630780 trunc from 0x02941c to 0x02ccb9 (0x389e bytes) 000819 1767323890.640730 MIRROR_RESYNC 4 mirrors 000820 1767323891.244884 fallocate 0x02eb65 thru 0x03673c (0x7bd8 bytes) 000821 1767323891.277164 MIRROR_RESYNC 4 mirrors 000822 1767323891.910871 trunc from 0x00357d to 0x00935f (0x5de3 bytes) 000823 1767323891.941657 read 0x00cb24 thru 0x01be73 (0xf350 bytes) 000824 1767323891.949311 mapread 0x02c8c1 thru 0x03345b (0x6b9b bytes) 000825 1767323891.960493 read 0x0051ba thru 0x008131 (0x2f78 bytes) 000826 1767323891.967513 punch from 0x002b88 to 0x0082e7 (0x5760 bytes) 000827 1767323891.970849 trunc from 0x040000 to 0x00fad8 (0x30527 bytes) 000828 1767323891.983367 punch from 0x00bf71 to 0x00ef6c (0x2ffc bytes) 000829 1767323891.987366 fallocate 0x01966c thru 0x027b47 (0xe4dc bytes) 000830 1767323891.991859 punch from 0x01b7a0 to 0x022b49 (0x73aa bytes) 000831 1767323891.995904 trunc from 0x027b48 to 0x02c376 (0x482f bytes) 000832 1767323892.016315 trunc from 0x02c377 to 0x00b33b (0x2103b bytes) 000833 1767323892.027674 trunc from 0x003115 to 0x00b33b (0x8227 bytes) 000834 1767323892.031722 trunc from 0x00b33c to 0x02e8f6 (0x235bb bytes) 000835 1767323892.047804 mapread 0x021903 thru 0x028031 (0x672f bytes) 000836 1767323892.075955 punch from 0x005bdd to 0x00da0f (0x7e33 bytes) 000837 1767323892.078937 read 0x026821 thru 0x02e8f6 (0x80d6 bytes) 000838 1767323892.084675 punch from 0x023cd8 to 0x028b6e (0x4e97 bytes) 000839 1767323892.090152 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2e8f7): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e8f7): Mirror 11: 0xd7a13ec3 (stale) Mirror 12: 0xd7a13ec3 (stale) Mirror 13: 0xd7a13ec3 (stale) CRC-32 checksum value for chunk [0, 0x2e8f7): Mirror 1: 0x57c21003 Mirror 14: 0x57c21003 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e8f7: skipped 000840 1767323892.600704 read 0x00d5fd thru 0x018e38 (0xb83c bytes) 000841 1767323892.609382 write 0x02a7a0 thru 0x039836 (0xf097 bytes) 000842 1767323892.634291 mapread 0x016fe7 thru 0x01a477 (0x3491 bytes) 000843 1767323892.641401 read 0x0164a0 thru 0x0222af (0xbe10 bytes) 000844 1767323892.647575 write 0x021687 thru 0x02343e (0x1db8 bytes) 000845 1767323892.648591 punch from 0x017f0f to 0x01ddf1 (0x5ee3 bytes) 000846 1767323892.652453 read 0x02535e thru 0x031ba5 (0xc848 bytes) 000847 1767323892.652989 read 0x0329c2 thru 0x039836 (0x6e75 bytes) 000848 1767323892.653865 trunc from 0x039837 to 0x031250 (0x85e6 bytes) 000849 1767323892.688774 write 0x036d97 thru 0x03ffff (0x9269 bytes) 000850 1767323892.690058 write 0x018620 thru 0x01e6dc (0x60bd bytes) 000851 1767323892.697283 trunc from 0x040000 to 0x01cf07 (0x230f8 bytes) 000852 1767323892.716122 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1cf08): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cf08): Mirror 11: 0xcd161bfc (stale) Mirror 12: 0xcd161bfc (stale) Mirror 13: 0xcd161bfc (stale) Mirror 14: 0x84c493a0 (stale) CRC-32 checksum value for chunk [0, 0x1cf08): Mirror 1: 0xc684d2f3 Mirror 15: 0xc684d2f3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cf08: skipped 000853 1767323893.175456 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x303:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1cf08): Mirror 12: 0xcd161bfc (stale) Mirror 13: 0xcd161bfc (stale) Mirror 14: 0x84c493a0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cf08): Mirror 12: 0xcd161bfc (stale) Mirror 13: 0xcd161bfc (stale) Mirror 14: 0x84c493a0 (stale) CRC-32 checksum value for chunk [0, 0x1cf08): Mirror 1: 0xc684d2f3 Mirror 15: 0xc684d2f3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cf08: skipped 000854 1767323893.654352 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1cf08): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cf08): Mirror 12: 0xcd161bfc (stale) Mirror 13: 0xcd161bfc (stale) Mirror 14: 0x84c493a0 (stale) CRC-32 checksum value for chunk [0, 0x1cf08): Mirror 1: 0xc684d2f3 Mirror 15: 0xc684d2f3 Mirror 16: 0xc684d2f3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cf08: skipped 000855 1767323894.258312 trunc from 0x004474 to 0x004686 (0x213 bytes) 000856 1767323894.296904 mapread 0x01892c thru 0x01cf07 (0x45dc bytes) 000857 1767323894.317234 mapread 0x00ecdd thru 0x01b08f (0xc3b3 bytes) 000858 1767323894.327569 mapread 0x00e0ca thru 0x01a21b (0xc152 bytes) 000859 1767323894.328493 trunc from 0x000974 to 0x002070 (0x16fd bytes) 000861 1767323894.338061 punch from 0x013fab to 0x0149ff (0xa55 bytes) 000862 1767323894.345361 write 0x002a1d thru 0x003a6a (0x104e bytes) 000863 1767323894.354600 trunc from 0x01b65f to 0x01cf07 (0x18a9 bytes) 000864 1767323894.362224 read 0x00dba7 thru 0x01a15c (0xc5b6 bytes) 000865 1767323894.365463 fallocate 0x001682 thru 0x00ce41 (0xb7c0 bytes) 000866 1767323894.374814 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x303:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1cf08): Mirror 13: 0xcd161bfc (stale) Mirror 14: 0x84c493a0 (stale) Mirror 15: 0xc684d2f3 (stale) Mirror 16: 0xc684d2f3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cf08): Mirror 13: 0xcd161bfc (stale) Mirror 14: 0x84c493a0 (stale) Mirror 15: 0xc684d2f3 (stale) Mirror 16: 0xc684d2f3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cf08: skipped 000867 1767323894.692663 read 0x00d7a6 thru 0x010c70 (0x34cb bytes) 000868 1767323894.697332 write 0x0322bb thru 0x03faac (0xd7f2 bytes) 000869 1767323894.704317 trunc from 0x03faad to 0x019345 (0x26767 bytes) 000870 1767323894.730941 trunc from 0x015d4c to 0x019345 (0x35fa bytes) 000871 1767323894.734621 mapwrite 0x0258dd thru 0x0355a5 (0xfcc9 bytes) 000873 1767323894.786997 trunc from 0x0355a6 to 0x0313bd (0x41e8 bytes) 000874 1767323894.808973 read 0x00d052 thru 0x0171c2 (0xa171 bytes) 000875 1767323894.823858 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x313be): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x313be): Mirror 13: 0x8b37b796 (stale) Mirror 14: 0x592f53cf (stale) Mirror 15: 0xfc1f35a7 (stale) Mirror 16: 0xfc1f35a7 (stale) CRC-32 checksum value for chunk [0, 0x313be): Mirror 1: 0xe4e400f0 Mirror 17: 0xe4e400f0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x313be: skipped 000876 1767323895.624271 mapread 0x02950f thru 0x02a9c6 (0x14b8 bytes) 000877 1767323895.649182 fallocate 0x01f317 thru 0x02e84d (0xf537 bytes) 000878 1767323895.682842 punch from 0x0174b7 to 0x0261cb (0xed15 bytes) 000879 1767323895.691458 MIRROR_RESYNC 6 mirrors 000880 1767323896.555127 trunc from 0x0313be to 0x02fd92 (0x162b bytes) 000881 1767323896.604954 mapread 0x012112 thru 0x01c9ea (0xa8d9 bytes) 000882 1767323896.618037 read 0x024860 thru 0x02fd92 (0xb533 bytes) 000883 1767323896.629362 mapread 0x005492 thru 0x011cfd (0xc86c bytes) 000884 1767323896.636587 punch from 0x0073e3 to 0x00abba (0x37d8 bytes) 000885 1767323896.639938 mapread 0x001e38 thru 0x00d68a (0xb853 bytes) 000886 1767323896.647955 mapread 0x00c29d thru 0x012338 (0x609c bytes) 000888 1767323896.650008 mapwrite 0x00f09b thru 0x00fd51 (0xcb7 bytes) 000889 1767323896.662766 punch from 0x01c47a to 0x02ba47 (0xf5ce bytes) 000890 1767323896.667189 write 0x027e4d thru 0x02f762 (0x7916 bytes) 000891 1767323896.672172 fallocate 0x035f90 thru 0x03b004 (0x5075 bytes) 000892 1767323896.676837 read 0x02e64a thru 0x02fd92 (0x1749 bytes) 000893 1767323896.677894 punch from 0x00ffc4 to 0x0178b4 (0x78f1 bytes) 000894 1767323896.685484 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x303:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fd93): Mirror 13: 0x1f436999 (stale) Mirror 15: 0x1f436999 (stale) Mirror 16: 0x1f436999 (stale) Mirror 17: 0x1f436999 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fd93): Mirror 13: 0x1f436999 (stale) Mirror 15: 0x1f436999 (stale) Mirror 16: 0x1f436999 (stale) Mirror 17: 0x1f436999 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fd93: skipped 000895 1767323896.965014 mapwrite 0x0211f4 thru 0x02ffae (0xedbb bytes) 000896 1767323897.029652 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x303:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ffaf): Mirror 13: 0x9555379f (stale) Mirror 15: 0x9555379f (stale) Mirror 17: 0x9555379f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ffaf): Mirror 13: 0x9555379f (stale) Mirror 15: 0x9555379f (stale) Mirror 17: 0x9555379f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ffaf: skipped 000897 1767323897.227515 mapread 0x00cc45 thru 0x00f73e (0x2afa bytes) 000898 1767323897.238947 punch from 0x00d129 to 0x01cedd (0xfdb5 bytes) 000899 1767323897.243914 read 0x018e69 thru 0x02866b (0xf803 bytes) 000901 1767323897.254212 mapwrite 0x035d45 thru 0x03d61e (0x78da bytes) 000902 1767323897.295920 fallocate 0x01f5bc thru 0x02f34a (0xfd8f bytes) 000904 1767323897.300644 trunc from 0x00b6f1 to 0x01a61a (0xef2a bytes) 000906 1767323897.305498 read 0x03c708 thru 0x03d61e (0xf17 bytes) 000907 1767323897.306156 read 0x0091fc thru 0x009539 (0x33e bytes) 000908 1767323897.307002 read 0x01c849 thru 0x0218c8 (0x5080 bytes) 000909 1767323897.307852 mapread 0x0308df thru 0x03b240 (0xa962 bytes) 000910 1767323897.309344 mapread 0x031583 thru 0x0357c2 (0x4240 bytes) 000911 1767323897.310173 fallocate 0x0301ab thru 0x03ffff (0xfe55 bytes) 000912 1767323897.316152 fallocate 0x020aaa thru 0x02c36e (0xb8c5 bytes) 000913 1767323897.325496 MIRROR_RESYNC 4 mirrors 000914 1767323898.053129 fallocate 0x02a009 thru 0x0367a4 (0xc79c bytes) 000915 1767323898.094924 trunc from 0x03d61f to 0x00092b (0x3ccf3 bytes) 000916 1767323898.131621 trunc from 0x00032f to 0x00092b (0x5fd bytes) 000917 1767323898.137444 mapread 0x000154 thru 0x00092b (0x7d8 bytes) 000918 1767323898.143127 write 0x03d885 thru 0x03ffff (0x277b bytes) 000919 1767323898.143983 fallocate 0x00de1a thru 0x01cee1 (0xf0c8 bytes) 000920 1767323898.147150 punch from 0x009cfc to 0x00bec9 (0x21ce bytes) 000921 1767323898.150127 write 0x020dbd thru 0x0247b1 (0x39f5 bytes) 000922 1767323898.156365 punch from 0x0291f5 to 0x032732 (0x953e bytes) 000923 1767323898.159176 trunc from 0x040000 to 0x01545f (0x2aba0 bytes) 000924 1767323898.171711 trunc from 0x00b2e5 to 0x01545f (0xa17b bytes) 000925 1767323898.174638 read 0x00781f thru 0x00a15f (0x2941 bytes) 000927 1767323898.180602 mapread 0x0133cd thru 0x0150e6 (0x1d1a bytes) 000928 1767323898.187643 punch from 0x014934 to 0x0151c6 (0x893 bytes) 000929 1767323898.197196 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x303:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15460): Mirror 15: 0x8b2a010d (stale) Mirror 17: 0x8b2a010d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15460): Mirror 15: 0x8b2a010d (stale) Mirror 17: 0x8b2a010d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15460: skipped 000930 1767323898.459934 fallocate 0x0262a7 thru 0x02c2ea (0x6044 bytes) 000931 1767323898.464020 read 0x023ef8 thru 0x02596a (0x1a73 bytes) 000932 1767323898.469966 trunc from 0x02c2eb to 0x03a85c (0xe572 bytes) 000933 1767323898.493403 trunc from 0x03a85d to 0x0335bc (0x72a0 bytes) 000934 1767323898.507755 punch from 0x00bc6e to 0x015c5a (0x9fed bytes) 000935 1767323898.517967 mapwrite 0x00d514 thru 0x01ad1b (0xd808 bytes) 000936 1767323898.549698 mapwrite 0x005333 thru 0x01304f (0xdd1d bytes) 000937 1767323898.580504 read 0x02b5a2 thru 0x03338b (0x7dea bytes) 000938 1767323898.589691 read 0x02e983 thru 0x031099 (0x2717 bytes) 000939 1767323898.591497 trunc from 0x013cca to 0x0211b6 (0xd4ed bytes) 000940 1767323898.598017 mapwrite 0x009b32 thru 0x00b439 (0x1908 bytes) 000941 1767323898.620312 trunc from 0x0335bd to 0x00b081 (0x2853b bytes) 000942 1767323898.664554 mapwrite 0x01c1ba thru 0x021b36 (0x597d bytes) 000943 1767323898.720895 mapread 0x00f348 thru 0x0120dc (0x2d95 bytes) 000944 1767323898.725218 mapwrite 0x00d5c5 thru 0x0106fa (0x3136 bytes) 000945 1767323898.740018 mapread 0x01f7fb thru 0x021b36 (0x233c bytes) 000946 1767323898.740809 read 0x00947f thru 0x011b41 (0x86c3 bytes) 000947 1767323898.741670 trunc from 0x018b54 to 0x01ad77 (0x2224 bytes) 000949 1767323898.745476 mapwrite 0x02c590 thru 0x02ebcc (0x263d bytes) 000951 1767323898.787512 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x303:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ebcd): Mirror 15: 0x6a59efcd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ebcd): Mirror 15: 0x6a59efcd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ebcd: skipped 000952 1767323899.001676 read 0x02def4 thru 0x02ebcc (0xcd9 bytes) 000953 1767323899.005582 write 0x037d18 thru 0x03ffff (0x82e8 bytes) 000954 1767323899.011957 read 0x01ded5 thru 0x024c93 (0x6dbf bytes) 000955 1767323899.024880 read 0x01282b thru 0x0165b7 (0x3d8d bytes) 000956 1767323899.025406 write 0x02c47b thru 0x0315d3 (0x5159 bytes) 000957 1767323899.026743 write 0x00ab3b thru 0x00c17e (0x1644 bytes) 000958 1767323899.031266 mapwrite 0x017cf6 thru 0x0272cb (0xf5d6 bytes) 000959 1767323899.058897 write 0x004e8c thru 0x00e008 (0x917d bytes) 000960 1767323899.060071 read 0x03a63d thru 0x03ffff (0x59c3 bytes) 000961 1767323899.060516 write 0x021ee5 thru 0x02b86e (0x998a bytes) 000962 1767323899.061443 write 0x029605 thru 0x0357c0 (0xc1bc bytes) 000963 1767323899.062284 trunc from 0x040000 to 0x001820 (0x3e7df bytes) 000964 1767323899.076224 trunc from 0x001821 to 0x01e364 (0x1cb44 bytes) 000965 1767323899.085913 punch from 0x0111ab to 0x01e364 (0xd1ba bytes) 000966 1767323899.088791 punch from 0x002e7e to 0x00aebc (0x803f bytes) 000967 1767323899.091230 mapread 0x006011 thru 0x00b414 (0x5404 bytes) 000968 1767323899.096355 fallocate 0x033bad thru 0x038e8c (0x52e0 bytes) 000969 1767323899.099116 mapwrite 0x004443 thru 0x013e13 (0xf9d1 bytes) 000970 1767323899.119972 fallocate 0x0179ce thru 0x026686 (0xecb9 bytes) 000971 1767323899.127067 read 0x003ef0 thru 0x0134f8 (0xf609 bytes) 000972 1767323899.127600 trunc from 0x00a32c to 0x00d6d4 (0x33a9 bytes) 000973 1767323899.131978 punch from 0x009f34 to 0x00ae93 (0xf60 bytes) 000974 1767323899.136460 write 0x028d53 thru 0x033342 (0xa5f0 bytes) 000975 1767323899.139733 mapwrite 0x02d7f4 thru 0x030d71 (0x357e bytes) 000976 1767323899.162781 write 0x03f2ac thru 0x03ffff (0xd54 bytes) 000977 1767323899.163745 mapread 0x0381de thru 0x03ffff (0x7e22 bytes) 000978 1767323899.172993 write 0x032480 thru 0x032695 (0x216 bytes) 000979 1767323899.174153 trunc from 0x03ca07 to 0x03ffff (0x35f9 bytes) 000980 1767323899.191391 MIRROR_RESYNC 2 mirrors 000981 1767323899.628483 MIRROR_RESYNC 2 mirrors 000982 1767323899.887787 MIRROR_EXTEND to 3 mirrors 000983 1767323900.463373 mapread 0x025d4d thru 0x026f00 (0x11b4 bytes) 000984 1767323900.472251 write 0x02ba58 thru 0x0319c8 (0x5f71 bytes) 000985 1767323900.515426 write 0x028c18 thru 0x0338c0 (0xaca9 bytes) 000986 1767323900.516733 write 0x03ce66 thru 0x03ff1d (0x30b8 bytes) 000987 1767323900.537794 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x303:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xcc177d81 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xcc177d81 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000988 1767323900.803601 write 0x002eb4 thru 0x00ba9c (0x8be9 bytes) 000989 1767323900.817451 trunc from 0x03a2f8 to 0x03ffff (0x5d08 bytes) 000990 1767323900.822988 trunc from 0x023655 to 0x030c7c (0xd628 bytes) 000991 1767323900.827365 mapread 0x020189 thru 0x024aa3 (0x491b bytes) 000992 1767323900.835704 write 0x039a21 thru 0x03b29e (0x187e bytes) 000993 1767323900.843207 write 0x030dd1 thru 0x03ade0 (0xa010 bytes) 000994 1767323900.854045 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xcc177d81 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb670c83e Mirror 17: 0xb670c83e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000995 1767323901.692980 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xcc177d81 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb670c83e Mirror 17: 0xb670c83e Mirror 18: 0xb670c83e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000996 1767323902.568479 trunc from 0x040000 to 0x01398a (0x2c675 bytes) 000997 1767323902.621620 fallocate 0x01d9d9 thru 0x01ea9d (0x10c5 bytes) 000998 1767323902.628064 fallocate 0x03da7b thru 0x03ffff (0x2585 bytes) 000999 1767323902.638350 punch from 0x01d1c9 to 0x01ea9d (0x18d5 bytes) 001000 1767323902.648311 punch from 0x0192aa to 0x01ea9d (0x57f4 bytes) All operations completed A-OK! PASS 70a (51s) == sanity-flr test 71: check mirror extend parent fid ==== 22:18:25 (1767323905) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000404:0x33d:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (7s) == sanity-flr test 72: check append on FLR file ========== 22:18:32 (1767323912) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.154509 s, 13.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077094 s, 13.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8c:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (5s) == sanity-flr test 200a: stress test ===================== 22:18:37 (1767323917) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8937696 .. Extending file size to 2533856 .. done Extending file size to 5148320 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5751968 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5989344 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7945888 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8684096 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8892544 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9231424 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 63234 63235 63237 63238 63239 done 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) 192.168.202.128@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (70s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 22:19:48 (1767323988) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8752256 .. done Extending file size to 8134688 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9074560 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8757984 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8941248 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9199520 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 65826 65827 65828 65829 65831 done 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) 192.168.202.128@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (71s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 22:20:58 (1767324058) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2424 1268880 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2356 1268948 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24672 3582348 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18896 3588124 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43568 7170472 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2424 1268880 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2356 1268948 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24672 3582348 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18896 3588124 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43568 7170472 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0506879 s, 41.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0886446 s, 23.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x83f4a488 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) PASS 200c (10s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 22:21:09 (1767324069) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0701465 s, 29.9 MB/s PASS 202 (5s) == sanity-flr test 203: mirror file preserve mirror ID === 22:21:14 (1767324074) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3a:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa3:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa3:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa4:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xa1:0x0] } PASS 203 (5s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 22:21:19 (1767324079) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0663746 s, 31.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (4s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 22:21:23 (1767324083) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0816597 s, 25.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa5:0x0] } PASS 204b (6s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 22:21:29 (1767324089) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0636862 s, 32.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa7:0x0] } PASS 204c (28s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 22:21:57 (1767324117) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg228-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565683 s, 18.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x411:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x415:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x414:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x411:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x415:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x414:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x412:0x0] } PASS 204d (41s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 22:22:38 (1767324158) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg228-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0730289 s, 28.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (35s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 22:23:13 (1767324193) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg228-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg228-server: Pool lustre.test_204f created Adding targets to pool oleg228-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg228-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0688518 s, 30.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg228-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg228-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg228-server: Pool lustre.test_204f destroyed PASS 204f (52s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 22:24:05 (1767324245) PASS 205a (5s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 22:24:10 (1767324250) PASS 205b (4s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 22:24:14 (1767324254) oleg228-server: Pool lustre.test_206 created Adding targets to pool oleg228-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg228-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg228-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg228-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg228-server: Pool lustre.test_206 destroyed PASS 206 (20s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 22:24:34 (1767324274) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642699 s, 16.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171947 s, 61.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x1c9fb39e Mirror 3: 0x1c9fb39e PASS 207 (6s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 22:24:40 (1767324280) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 22:24:42 (1767324282) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 22:24:44 (1767324284) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.042055 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1ab0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1ca0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1ca0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1ca0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1f80000 Read this (4 bytes): BBBB 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) PASS 209a (22s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 22:25:07 (1767324307) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0145839 s, 562 kB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 22:25:10 (1767324310) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236299 s, 44.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5d lmm_fid: [0x200000405:0x5d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5d lmm_fid: [0x200000405:0x5d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb4:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611229 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-01 22:25:11.000000000 -0500 Modify: 2026-01-01 22:25:11.000000000 -0500 Change: 2026-01-01 22:25:12.000000000 -0500 Birth: 2026-01-01 22:25:12.000000000 -0500 PASS 210a (5s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 22:25:15 (1767324315) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265263 s, 39.5 MB/s oleg228-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg228-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg228-server: O/280000401/d21/b5: File not found by ext2_lookup PASS 210b (17s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 22:25:32 (1767324332) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.138284 s, 296 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0174906 s, 234 kB/s size after second write -rw-r--r-- 1 root root 4096 Jan 1 22:25 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x62:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jan 1 22:25 /mnt/lustre/f211.sanity-flr PASS 211 (5s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 22:25:37 (1767324337) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (5s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 22:25:42 (1767324342) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (5s) == sanity-flr test complete, duration 1213 sec =========== 22:25:47 (1767324347) === sanity-flr: start cleanup 22:25:49 (1767324349) === === sanity-flr: finish cleanup 22:25:55 (1767324355) ===