-----============= acceptance-small: sanity-flr ============----- Thu Jan 22 04:45:52 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg234-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 04:46:14 (1769075174) === oleg234-client.virtnet: executing check_config_client /mnt/lustre oleg234-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg234-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9bbce0032800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9bbce0032800.idle_timeout=debug disable quota as required oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 04:47:02 (1769075222) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8211] == sanity-flr test 0a: lfs mirror create with -N option == 04:47:05 (1769075225) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 04:47:25 (1769075245) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 04:47:29 (1769075249) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 04:47:33 (1769075253) PASS 0d (29s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 04:48:02 (1769075282) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 04:48:06 (1769075286) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 04:48:10 (1769075290) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9bbce0032800.stats=clear osc.lustre-OST0001-osc-ffff9bbce0032800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 04:48:19 (1769075299) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 04:48:30 (1769075310) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 04:48:39 (1769075319) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 04:48:43 (1769075323) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 04:48:55 (1769075335) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 04:49:03 (1769075343) PASS 3 (11s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 04:49:14 (1769075354) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (10s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 04:49:24 (1769075364) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 04:49:38 (1769075378) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 04:49:46 (1769075386) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 1.58837 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190618 s, 5.5 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 04:49:57 (1769075397) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.500222 s, 33.5 MB/s PASS 22 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 04:50:08 (1769075408) Stopping /mnt/lustre-ost1 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9bbce0032800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9bbce0032800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg234-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9bbce0032800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9bbce0032800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg234-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (89s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 04:51:37 (1769075497) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.552666 s, 13.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9bbce0032800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9bbce0032800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg234-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (49s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 04:52:26 (1769075546) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9bbce0032800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9bbce0032800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg234-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9bbce0032800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9bbce0032800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg234-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (90s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 04:53:57 (1769075637) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.41989 s, 15.3 MB/s -rw-r--r-- 1 root root 52428800 Jan 22 04:54 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.759s user 0m0.000s sys 0m0.216s Stopping /mnt/lustre-ost1 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9bbce0032800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9bbce0032800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.400s user 0m0.001s sys 0m0.236s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg234-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9bbce0032800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9bbce0032800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.617s user 0m0.001s sys 0m0.189s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg234-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg234-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9bbce0032800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9bbce0032800.ost_server_uuid in FULL state after 0 sec PASS 33b (102s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 04:55:39 (1769075739) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 04:55:43 (1769075743) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 04:55:46 (1769075746) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 04:55:50 (1769075750) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 04:55:59 (1769075759) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.711595 s, 17.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 04:56:14 (1769075774) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259168 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277311 s, 37.8 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 04:56:23 (1769075783) PASS 36c (12s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 04:56:35 (1769075795) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00934715 s, 110 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00162065 s, 632 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000821908 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00207736 s, 493 kB/s flv=10 olv=10 PASS 36d (16s) == sanity-flr test 37: mirror I/O API verification ======= 04:56:51 (1769075811) create test files with size 15747813 .. 618bfef7106db65b3b074be2d58911b1 - 824276848b359068e1fc47f2d9833be7 - 0b5105fa840d932efd1840f8f9b31541 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg234-client.virtnet /mnt/lustre (opts:) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre PASS 37 (32s) == sanity-flr test 38: resync ============================ 04:57:23 (1769075843) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 04:57:38 (1769075858) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 04:57:46 (1769075866) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965231 s, 10.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230535 s, 45.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3d:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 04:57:54 (1769075874) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0903669 s, 23.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.4464 s, 9.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.148525 s, 14.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (21s) == sanity-flr test 42: lfs mirror verify ================= 04:58:15 (1769075895) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 04:58:20 (1769075900) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 04:58:24 (1769075904) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413918 s, 25.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 04:58:34 (1769075914) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.41907 s, 7.5 MB/s PASS 44a (13s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 04:58:47 (1769075927) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x48:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x44:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x44:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 04:59:00 (1769075940) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.436061 s, 24.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg234-client.virtnet /mnt/lustre (opts:) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 04:59:14 (1769075954) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9bbcc55e2000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.516874 s, 20.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg234-client.virtnet /mnt/lustre (opts:) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9bbcc5349800.xattr_cache=1 PASS 44e (20s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 04:59:34 (1769075974) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (46s) == sanity-flr test 46: Verify setstripe --copy option ==== 05:00:21 (1769076021) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (50s) == sanity-flr test 47: Verify mirror obj alloc =========== 05:01:11 (1769076071) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 05:01:15 (1769076075) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.381409 s, 8.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.205199 s, 15.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xaf37d53a (stale) Mirror 2: 0xaf37d53a (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 5df3dd3a157b1133c5cb035f29a08fb2 - mirror 1 checksum: 6549e33a27c33f79fbf2416750fbee44 - mirror 2 checksum: 5df3dd3a157b1133c5cb035f29a08fb2 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jan 22 05:01:19 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jan 22 05:01:19 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 5df3dd3a157b1133c5cb035f29a08fb2 - mirror 1 checksum: 6549e33a27c33f79fbf2416750fbee44 - mirror 2 checksum: 6549e33a27c33f79fbf2416750fbee44 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jan 22 05:01:19 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jan 22 05:01:19 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 05:01:27 (1769076087) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734223 s, 14.3 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (10s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 05:01:37 (1769076097) Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 05:01:52 (1769076112) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 05:02:27 (1769076147) targets are mounted 05:02:27 (1769076147) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795679 s, 13.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } PASS 50A (72s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 05:02:50 (1769076170) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0391572 s, 523 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.025466 s, 804 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (16s) == sanity-flr test 50b: mirror rsync handles sparseness == 05:03:05 (1769076185) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.100945 s, 203 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.013487 s, 1.5 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 05:03:17 (1769076197) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0213915 s, 766 kB/s PASS 50c (11s) == sanity-flr test 50d: mirror rsync keep holes ========== 05:03:27 (1769076207) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0719332 s, 285 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (10s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 05:03:37 (1769076217) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00779201 s, 526 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0229812 s, 178 kB/s PASS 60a (12s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 05:03:49 (1769076229) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 05:04:14 (1769076254) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 05:04:45 (1769076285) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 05:05:09 (1769076309) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 05:05:36 (1769076336) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (11s) == sanity-flr test 70: mirror create and split race ====== 05:05:48 (1769076348) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 55985 mirror split pid 55986 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 55985 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 55986 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 05:06:57 (1769076417) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 6420 truncating to largest ever: 0x2c3 000001 1769076420.480124 trunc from 00000000 to 0x0002c2 (0x2c3 bytes) 000003 1769076420.500107 write 0x03b409 thru 0x03df65 (0x2b5d bytes) 000004 1769076420.501291 mapwrite 0x00cb2e thru 0x00e5c9 (0x1a9c bytes) 000005 1769076420.548162 write 0x0070cf thru 0x011b50 (0xaa82 bytes) 000007 1769076420.550848 write 0x012df7 thru 0x01f8e6 (0xcaf0 bytes) 000008 1769076420.565063 mapread 0x034e41 thru 0x03df65 (0x9125 bytes) 000009 1769076420.581705 write 0x00db49 thru 0x00fcc8 (0x2180 bytes) fallocating to largest ever: 0x1323d 000010 1769076420.583510 fallocate 0x00a93f thru 0x01323c (0x88fe bytes) 000011 1769076420.590591 punch from 0x0115f4 to 0x013e21 (0x282e bytes) 000012 1769076420.627687 MIRROR_EXTEND to 2 mirrors 000014 1769076421.719522 MIRROR_EXTEND to 3 mirrors 000015 1769076422.838261 read 0x00cbef thru 0x016233 (0x9645 bytes) 000016 1769076422.856695 punch from 0x01a302 to 0x01bcb8 (0x19b7 bytes) 000018 1769076422.891085 trunc from 0x02a41f to 0x0366a6 (0xc288 bytes) 000019 1769076422.897741 trunc from 0x0288c5 to 0x034320 (0xba5c bytes) 000020 1769076422.907753 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3df66): Mirror 3: 0xec8cc1c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3df66): Mirror 3: 0xec8cc1c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3df66: skipped 000021 1769076423.217975 mapwrite 0x017d9d thru 0x023b3c (0xbda0 bytes) 000022 1769076423.265692 mapread 0x01de07 thru 0x01ecae (0xea8 bytes) 000023 1769076423.267583 punch from 0x036728 to 0x03a684 (0x3f5d bytes) 000024 1769076423.272689 mapwrite 0x02d93f thru 0x02ee17 (0x14d9 bytes) 000025 1769076423.291704 punch from 0x0306a8 to 0x036303 (0x5c5c bytes) 000027 1769076423.300950 read 0x0282fd thru 0x032328 (0xa02c bytes) 000029 1769076423.310992 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3df66): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3df66): Mirror 3: 0xec8cc1c4 (stale) CRC-32 checksum value for chunk [0, 0x3df66): Mirror 1: 0x48be3430 Mirror 4: 0x48be3430 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3df66: skipped 000030 1769076424.473000 mapwrite 0x00e2d2 thru 0x0129ae (0x46dd bytes) 000031 1769076424.607462 mapwrite 0x00b7bc thru 0x019d87 (0xe5cc bytes) 000032 1769076424.659083 MIRROR_RESYNC 3 mirrors truncating to largest ever: 0x346a6 000033 1769076425.757427 trunc from 0x03df66 to 0x0346a5 (0x98c0 bytes) 000034 1769076425.810270 punch from 0x000e7f to 0x006e06 (0x5f88 bytes) 000035 1769076425.818470 fallocate 0x032245 thru 0x03446e (0x222a bytes) 000036 1769076425.825661 punch from 0x00abb5 to 0x0140b8 (0x9504 bytes) 000037 1769076425.832829 write 0x016c52 thru 0x017318 (0x6c7 bytes) 000038 1769076425.845531 write 0x03f2a1 thru 0x03ffff (0xd5f bytes) fallocating to largest ever: 0x39d37 000040 1769076425.848691 fallocate 0x02b753 thru 0x039d36 (0xe5e4 bytes) 000041 1769076425.860649 MIRROR_RESYNC 3 mirrors 000042 1769076426.980416 mapwrite 0x009915 thru 0x00c7f9 (0x2ee5 bytes) 000043 1769076427.144132 trunc from 0x034d18 to 0x03906d (0x4356 bytes) 000044 1769076427.151410 mapwrite 0x00f206 thru 0x0194fc (0xa2f7 bytes) 000045 1769076427.180942 trunc from 0x026d3e to 0x033d42 (0xd005 bytes) 000046 1769076427.186001 mapwrite 0x0286d5 thru 0x030eb4 (0x87e0 bytes) 000047 1769076427.237266 punch from 0x03e93f to 0x03ffff (0x16c1 bytes) 000048 1769076427.247299 write 0x020da6 thru 0x023105 (0x2360 bytes) 000049 1769076427.267474 punch from 0x03cafa to 0x03ffff (0x3506 bytes) 000050 1769076427.273667 mapwrite 0x0397c4 thru 0x03ffff (0x683c bytes) 000051 1769076427.306318 write 0x0102e6 thru 0x0105ba (0x2d5 bytes) 000052 1769076427.307686 read 0x00d597 thru 0x01d0b9 (0xfb23 bytes) 000053 1769076427.309493 mapread 0x01986c thru 0x022d63 (0x94f8 bytes) 000054 1769076427.310661 trunc from 0x040000 to 0x02bec7 (0x14138 bytes) 000055 1769076427.339059 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2bec8): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bec8): Mirror 3: 0xaed98157 (stale) Mirror 4: 0xaed98157 (stale) CRC-32 checksum value for chunk [0, 0x2bec8): Mirror 1: 0x86eb8eac Mirror 5: 0x86eb8eac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bec8: skipped 000057 1769076428.192685 fallocate 0x0094d8 thru 0x00d399 (0x3ec2 bytes) 000058 1769076428.240623 mapwrite 0x0156e5 thru 0x01fb2d (0xa449 bytes) 000060 1769076428.283665 trunc from 0x015661 to 0x023551 (0xdef1 bytes) 000061 1769076428.291246 fallocate 0x029302 thru 0x031f23 (0x8c22 bytes) 000062 1769076428.296854 trunc from 0x00f4b8 to 0x01901f (0x9b68 bytes) 000063 1769076428.304097 punch from 0x01e644 to 0x02bec7 (0xd884 bytes) 000064 1769076428.310653 mapwrite 0x0186c9 thru 0x026148 (0xda80 bytes) 000066 1769076428.399652 fallocate 0x02f259 thru 0x034541 (0x52e9 bytes) 000067 1769076428.408273 write 0x03d05f thru 0x03ffff (0x2fa1 bytes) 000068 1769076428.431729 write 0x00a426 thru 0x0167e5 (0xc3c0 bytes) 000069 1769076428.455113 fallocate 0x02a9b0 thru 0x02e481 (0x3ad2 bytes) 000070 1769076428.467013 read 0x018e99 thru 0x01f156 (0x62be bytes) fallocating to largest ever: 0x40000 000071 1769076428.482103 fallocate 0x03ff29 thru 0x03ffff (0xd7 bytes) 000072 1769076428.497428 fallocate 0x02e7c7 thru 0x03232a (0x3b64 bytes) 000073 1769076428.507725 trunc from 0x03cad2 to 0x03ffff (0x352e bytes) 000074 1769076428.538238 mapread 0x02c5c0 thru 0x031889 (0x52ca bytes) 000075 1769076428.562578 mapwrite 0x01cbd6 thru 0x01d903 (0xd2e bytes) 000076 1769076428.576835 trunc from 0x007839 to 0x0124dd (0xaca5 bytes) 000078 1769076428.582340 write 0x009a6a thru 0x00f3f3 (0x598a bytes) 000079 1769076428.592587 write 0x02666e thru 0x03020a (0x9b9d bytes) 000080 1769076428.593926 write 0x032f6e thru 0x03d0cb (0xa15e bytes) 000081 1769076428.601664 mapwrite 0x02a3ac thru 0x02f59a (0x51ef bytes) 000082 1769076428.625818 punch from 0x01062f to 0x01e5a9 (0xdf7b bytes) 000083 1769076428.635917 trunc from 0x040000 to 0x002623 (0x3d9dc bytes) 000084 1769076428.664587 mapwrite 0x01989f thru 0x022fd6 (0x9738 bytes) 000085 1769076428.745514 write 0x03ee87 thru 0x03ffff (0x1179 bytes) 000086 1769076428.746827 trunc from 0x02b69e to 0x031f16 (0x6879 bytes) 000087 1769076428.758650 trunc from 0x040000 to 0x01d14d (0x22eb2 bytes) 000088 1769076428.807026 read 0x01c920 thru 0x01d14d (0x82e bytes) 000089 1769076428.807757 trunc from 0x005a2f to 0x00ec4b (0x921d bytes) 000090 1769076428.821620 punch from 0x00682e to 0x00954c (0x2d1f bytes) 000091 1769076428.830966 mapread 0x007156 thru 0x00f2ca (0x8175 bytes) 000092 1769076428.844203 read 0x009ff9 thru 0x016550 (0xc558 bytes) 000094 1769076428.845360 trunc from 0x000e7a to 0x006fd2 (0x6159 bytes) 000095 1769076428.855107 fallocate 0x02c238 thru 0x02e062 (0x1e2b bytes) 000096 1769076428.861607 read 0x00d6ef thru 0x016c78 (0x958a bytes) 000097 1769076428.863042 trunc from 0x02e063 to 0x03ca50 (0xe9ee bytes) 000099 1769076428.885578 read 0x01e40b thru 0x01ecc0 (0x8b6 bytes) 000100 1769076428.904125 fallocate 0x033aca thru 0x03b805 (0x7d3c bytes) 000101 1769076428.916722 mapread 0x0282c6 thru 0x02dd46 (0x5a81 bytes) 000102 1769076428.920150 trunc from 0x021bb1 to 0x026ce8 (0x5138 bytes) 000103 1769076428.938319 punch from 0x011bfa to 0x01d064 (0xb46b bytes) 000104 1769076428.952260 trunc from 0x0026d6 to 0x00fcb2 (0xd5dd bytes) 000105 1769076428.983122 fallocate 0x018607 thru 0x01f02d (0x6a27 bytes) 000106 1769076428.998572 punch from 0x020f8d to 0x029560 (0x85d4 bytes) 000107 1769076429.018052 punch from 0x03c234 to 0x03ca50 (0x81d bytes) 000108 1769076429.036606 mapread 0x007163 thru 0x01371f (0xc5bd bytes) 000109 1769076429.234114 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ca51): Mirror 3: 0xbbc878ec (stale) Mirror 4: 0xbbc878ec (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ca51): Mirror 3: 0xbbc878ec (stale) Mirror 4: 0xbbc878ec (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ca51: skipped 000110 1769076429.616650 trunc from 0x02ce96 to 0x02dee9 (0x1054 bytes) 000111 1769076429.621763 mapread 0x032cc4 thru 0x038737 (0x5a74 bytes) 000113 1769076429.644987 read 0x0127e3 thru 0x01d918 (0xb136 bytes) 000114 1769076429.666055 trunc from 0x03ca51 to 0x01e565 (0x1e4eb bytes) 000115 1769076429.686136 write 0x00fe35 thru 0x0117d9 (0x19a5 bytes) 000116 1769076429.693856 write 0x025717 thru 0x02a7c1 (0x50ab bytes) 000117 1769076429.695598 trunc from 0x02a7c2 to 0x01cb95 (0xdc2c bytes) 000119 1769076429.711774 trunc from 0x01cb96 to 0x02766e (0xaad9 bytes) 000120 1769076429.727024 write 0x000e02 thru 0x00359f (0x279e bytes) 000121 1769076429.737620 fallocate 0x0213e6 thru 0x021668 (0x283 bytes) 000122 1769076429.747950 trunc from 0x02766f to 0x035e93 (0xe825 bytes) 000123 1769076429.787581 mapwrite 0x034322 thru 0x03ffff (0xbcde bytes) 000124 1769076429.850948 fallocate 0x0008cd thru 0x000bea (0x31e bytes) 000125 1769076429.865316 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2b0244a5 (stale) Mirror 4: 0x2b0244a5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf0da98e5 Mirror 5: 0xf0da98e5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000126 1769076431.043273 trunc from 0x040000 to 0x0249c9 (0x1b636 bytes) 000127 1769076431.100064 write 0x038eaf thru 0x03ffff (0x7151 bytes) 000128 1769076431.101477 mapread 0x03468c thru 0x037fa5 (0x391a bytes) 000129 1769076431.124849 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x2b0244a5 (stale) Mirror 5: 0xf0da98e5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x2b0244a5 (stale) Mirror 5: 0xf0da98e5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000131 1769076431.506775 mapwrite 0x00de3d thru 0x00f67c (0x1840 bytes) 000132 1769076431.539487 read 0x0176bd thru 0x025aa6 (0xe3ea bytes) 000133 1769076431.550346 fallocate 0x037ef2 thru 0x03dcb2 (0x5dc1 bytes) 000136 1769076431.565514 punch from 0x03d62a to 0x03ffff (0x29d6 bytes) 000139 1769076431.573586 mapwrite 0x008374 thru 0x00ee67 (0x6af4 bytes) 000140 1769076431.591539 trunc from 0x040000 to 0x009e3b (0x361c4 bytes) 000141 1769076431.617604 fallocate 0x038037 thru 0x03a22b (0x21f5 bytes) 000143 1769076431.624017 write 0x022756 thru 0x024c6d (0x2518 bytes) 000144 1769076431.630988 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24c6e): Mirror 5: 0xc4d26c06 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24c6e): Mirror 5: 0xc4d26c06 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24c6e: skipped 000145 1769076432.059635 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x24c6e): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24c6e): Mirror 5: 0xc4d26c06 (stale) CRC-32 checksum value for chunk [0, 0x24c6e): Mirror 1: 0x82139fa7 Mirror 6: 0x82139fa7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24c6e: skipped 000146 1769076432.999525 mapread 0x0008d8 thru 0x009f3d (0x9666 bytes) 000147 1769076433.052584 trunc from 0x024c6e to 0x0080d3 (0x1cb9a bytes) 000148 1769076433.161850 trunc from 0x006d5f to 0x0080d3 (0x1375 bytes) 000149 1769076433.182091 fallocate 0x005b96 thru 0x011a2c (0xbe97 bytes) 000150 1769076433.192009 mapwrite 0x03671e thru 0x03af25 (0x4808 bytes) 000151 1769076433.315769 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3af26): Mirror 5: 0x4d40996c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3af26): Mirror 5: 0x4d40996c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3af26: skipped 000152 1769076433.618699 write 0x0130f7 thru 0x0159db (0x28e5 bytes) 000153 1769076433.635559 punch from 0x020ade to 0x02c149 (0xb66c bytes) 000154 1769076433.641633 trunc from 0x03af26 to 0x00f13d (0x2bde8 bytes) 000155 1769076433.660640 mapread 0x00833e thru 0x00e247 (0x5f0a bytes) 000156 1769076433.671790 trunc from 0x0074f6 to 0x00f13d (0x7c48 bytes) 000157 1769076433.681497 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xf13e): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf13e): Mirror 5: 0x51593aec (stale) CRC-32 checksum value for chunk [0, 0xf13e): Mirror 1: 0x51593aec Mirror 6: 0x51593aec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf13e: skipped 000158 1769076435.101704 mapread 0x009abf thru 0x00b9b1 (0x1ef3 bytes) 000159 1769076435.141786 write 0x009a85 thru 0x00ae22 (0x139e bytes) 000160 1769076435.226063 trunc from 0x002839 to 0x00423f (0x1a07 bytes) 000161 1769076435.236034 mapwrite 0x028b67 thru 0x02dc50 (0x50ea bytes) 000162 1769076435.287008 trunc from 0x02dc51 to 0x038cd5 (0xb085 bytes) 000163 1769076435.306860 trunc from 0x038cd6 to 0x027698 (0x1163d bytes) 000164 1769076435.322662 mapread 0x01e6fe thru 0x0238a9 (0x51ac bytes) 000165 1769076435.328680 fallocate 0x021298 thru 0x0310aa (0xfe13 bytes) 000167 1769076435.333497 punch from 0x023f5a to 0x028493 (0x453a bytes) 000168 1769076435.341013 mapread 0x01c84d thru 0x025494 (0x8c48 bytes) 000169 1769076435.353765 trunc from 0x02f4b8 to 0x0310aa (0x1bf3 bytes) 000170 1769076435.360064 mapwrite 0x00b3fc thru 0x014b6d (0x9772 bytes) 000172 1769076435.381404 fallocate 0x03c65e thru 0x03fd97 (0x373a bytes) 000174 1769076435.388444 trunc from 0x028be2 to 0x037a8c (0xeeab bytes) 000175 1769076435.400758 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fd98): Mirror 5: 0x4dad9215 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd98): Mirror 5: 0x4dad9215 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd98: skipped 000176 1769076435.686809 write 0x0385b2 thru 0x03ffff (0x7a4e bytes) 000177 1769076435.709467 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xf0da98e5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7ae303d1 Mirror 6: 0x7ae303d1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000178 1769076436.567694 read 0x001418 thru 0x00e4f2 (0xd0db bytes) 000179 1769076436.583077 trunc from 0x040000 to 0x00c730 (0x338cf bytes) 000180 1769076436.646700 MIRROR_RESYNC 3 mirrors 000181 1769076437.383541 trunc from 0x00b69e to 0x00b6cf (0x32 bytes) 000182 1769076437.425369 trunc from 0x00b9d3 to 0x00c730 (0xd5e bytes) 000183 1769076437.431724 read 0x00406c thru 0x00c730 (0x86c5 bytes) 000184 1769076437.449675 MIRROR_RESYNC 3 mirrors 000185 1769076438.305566 punch from 0x005d72 to 0x00c730 (0x69bf bytes) 000186 1769076438.345466 write 0x01821e thru 0x01ad61 (0x2b44 bytes) 000188 1769076438.347863 fallocate 0x00e290 thru 0x01a501 (0xc272 bytes) 000189 1769076438.352910 mapwrite 0x008a13 thru 0x0161b4 (0xd7a2 bytes) 000190 1769076438.406195 trunc from 0x01ad62 to 0x010a59 (0xa308 bytes) 000191 1769076438.432803 write 0x025eac thru 0x029b3d (0x3c92 bytes) 000192 1769076438.436781 read 0x022cf1 thru 0x029b3d (0x6e4d bytes) 000193 1769076438.456181 MIRROR_RESYNC 3 mirrors 000195 1769076439.414868 punch from 0x021325 to 0x029b3d (0x8819 bytes) 000196 1769076439.463714 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x29b3e): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29b3e): Mirror 5: 0x2d782ca (stale) Mirror 6: 0x2d782ca (stale) CRC-32 checksum value for chunk [0, 0x29b3e): Mirror 1: 0x86931d25 Mirror 7: 0x86931d25 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29b3e: skipped 000197 1769076440.655015 mapwrite 0x03462b thru 0x038a85 (0x445b bytes) 000198 1769076440.792287 trunc from 0x038a86 to 0x00eab0 (0x29fd5 bytes) 000199 1769076440.835387 trunc from 0x00eab1 to 0x02490b (0x15e5b bytes) 000200 1769076440.876920 punch from 0x0073cb to 0x016088 (0xecbe bytes) 000201 1769076440.898774 MIRROR_RESYNC 4 mirrors 000202 1769076442.297587 trunc from 0x02490c to 0x024cf6 (0x3eb bytes) 000203 1769076442.371954 fallocate 0x03e065 thru 0x03ffff (0x1f9b bytes) 000204 1769076442.381818 read 0x01de1d thru 0x01e3c5 (0x5a9 bytes) 000205 1769076442.393172 trunc from 0x01ecdf to 0x02112c (0x244e bytes) 000206 1769076442.412632 fallocate 0x03452b thru 0x03860c (0x40e2 bytes) 000207 1769076442.429896 mapread 0x011cb5 thru 0x020665 (0xe9b1 bytes) 000210 1769076442.493788 trunc from 0x024cf7 to 0x02b276 (0x6580 bytes) 000211 1769076442.547762 mapread 0x026ffd thru 0x02b276 (0x427a bytes) 000212 1769076442.579698 mapwrite 0x00fe48 thru 0x01444c (0x4605 bytes) 000213 1769076442.631919 fallocate 0x016ef7 thru 0x01a6ea (0x37f4 bytes) 000214 1769076442.644084 fallocate 0x00f61c thru 0x00fb02 (0x4e7 bytes) 000216 1769076442.657846 read 0x026ffa thru 0x02b276 (0x427d bytes) 000217 1769076442.658516 write 0x0394d5 thru 0x03ffff (0x6b2b bytes) 000218 1769076442.660931 fallocate 0x00b7df thru 0x01afd3 (0xf7f5 bytes) 000219 1769076442.670455 trunc from 0x040000 to 0x0133b4 (0x2cc4b bytes) 000221 1769076442.714500 punch from 0x01186b to 0x0133b4 (0x1b4a bytes) 000222 1769076442.719598 fallocate 0x033d71 thru 0x039ea3 (0x6133 bytes) 000223 1769076442.731460 write 0x01703d thru 0x01922b (0x21ef bytes) 000224 1769076442.746678 fallocate 0x011c6e thru 0x0202b7 (0xe64a bytes) 000225 1769076442.755665 write 0x016e93 thru 0x022b1a (0xbc88 bytes) 000226 1769076442.760203 fallocate 0x003bfe thru 0x011621 (0xda24 bytes) 000228 1769076442.767819 write 0x01bd45 thru 0x01f074 (0x3330 bytes) 000230 1769076442.769853 trunc from 0x039ea4 to 0x02135a (0x18b49 bytes) 000231 1769076442.792029 read 0x00d2f0 thru 0x013040 (0x5d51 bytes) 000232 1769076442.799564 MIRROR_RESYNC 4 mirrors 000233 1769076444.045761 fallocate 0x03366c thru 0x03c2ec (0x8c81 bytes) 000234 1769076444.079268 punch from 0x002153 to 0x005c61 (0x3b0f bytes) 000235 1769076444.090690 read 0x0028a1 thru 0x009421 (0x6b81 bytes) 000236 1769076444.103665 trunc from 0x03c2ed to 0x0097b6 (0x32b36 bytes) 000237 1769076444.148558 trunc from 0x006e9d to 0x0097b6 (0x291a bytes) 000238 1769076444.158436 trunc from 0x0097b7 to 0x007979 (0x1e3d bytes) 000239 1769076444.177458 read 0x004a1e thru 0x007979 (0x2f5c bytes) 000240 1769076444.187590 write 0x0060e8 thru 0x01583f (0xf758 bytes) 000241 1769076444.189785 read 0x002957 thru 0x01060f (0xdcb9 bytes) 000242 1769076444.195545 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15840): Mirror 6: 0xd0049ebf (stale) Mirror 7: 0xd0049ebf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15840): Mirror 6: 0xd0049ebf (stale) Mirror 7: 0xd0049ebf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15840: skipped 000243 1769076444.593979 write 0x02ecba thru 0x039771 (0xaab8 bytes) 000244 1769076444.604517 mapwrite 0x0080af thru 0x009df5 (0x1d47 bytes) 000245 1769076444.645106 write 0x009b48 thru 0x00da6c (0x3f25 bytes) 000246 1769076444.662556 MIRROR_RESYNC 3 mirrors 000247 1769076445.720256 punch from 0x0298c3 to 0x02fca4 (0x63e2 bytes) 000248 1769076445.768510 mapwrite 0x0226ca thru 0x031df0 (0xf727 bytes) 000249 1769076445.837495 mapread 0x03537e thru 0x039771 (0x43f4 bytes) 000250 1769076445.847967 punch from 0x0004c5 to 0x0007a0 (0x2dc bytes) 000251 1769076445.856819 mapwrite 0x033019 thru 0x03ffff (0xcfe7 bytes) 000252 1769076445.933519 fallocate 0x00198a thru 0x005daa (0x4421 bytes) 000253 1769076445.940064 trunc from 0x040000 to 0x0170ce (0x28f31 bytes) 000254 1769076445.965838 read 0x007d10 thru 0x00c15f (0x4450 bytes) 000255 1769076445.973056 read 0x005cc7 thru 0x006084 (0x3be bytes) 000256 1769076445.985023 mapread 0x016560 thru 0x0170ce (0xb6f bytes) 000257 1769076445.990711 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x170cf): Mirror 7: 0x6d5e8b5c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x170cf): Mirror 7: 0x6d5e8b5c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x170cf: skipped 000258 1769076446.428164 fallocate 0x00f899 thru 0x010b46 (0x12ae bytes) 000259 1769076446.438352 mapwrite 0x0049b5 thru 0x0110fe (0xc74a bytes) 000260 1769076446.495004 MIRROR_RESYNC 2 mirrors 000261 1769076447.626412 fallocate 0x01bd15 thru 0x02ae2e (0xf11a bytes) 000262 1769076447.717840 fallocate 0x0380c0 thru 0x03ffff (0x7f40 bytes) 000263 1769076447.756461 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x170cf): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x170cf): Mirror 7: 0xd022899 (stale) CRC-32 checksum value for chunk [0, 0x170cf): Mirror 1: 0xd022899 Mirror 8: 0xd022899 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x170cf: skipped 000264 1769076449.197358 fallocate 0x004411 thru 0x01386a (0xf45a bytes) 000265 1769076449.245849 read 0x00dac9 thru 0x0111ac (0x36e4 bytes) 000266 1769076449.258197 read 0x016055 thru 0x0170ce (0x107a bytes) 000267 1769076449.269198 write 0x015fdb thru 0x01cb41 (0x6b67 bytes) 000268 1769076449.283369 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1cb42): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cb42): Mirror 7: 0x2ec3ee84 (stale) Mirror 8: 0x2ec3ee84 (stale) CRC-32 checksum value for chunk [0, 0x1cb42): Mirror 1: 0xdf2741e1 Mirror 9: 0xdf2741e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cb42: skipped 000269 1769076450.040201 write 0x022a9a thru 0x026e94 (0x43fb bytes) 000270 1769076450.131137 punch from 0x01389d to 0x01e71e (0xae82 bytes) 000272 1769076450.150499 punch from 0x024c3c to 0x026e94 (0x2259 bytes) 000273 1769076450.171198 mapread 0x01acb7 thru 0x023f1c (0x9266 bytes) 000275 1769076450.217492 trunc from 0x00f613 to 0x012b11 (0x34ff bytes) 000276 1769076450.232155 write 0x03b2f6 thru 0x03ffff (0x4d0a bytes) 000277 1769076450.233517 trunc from 0x014498 to 0x019a14 (0x557d bytes) 000278 1769076450.259987 MIRROR_RESYNC 4 mirrors 000279 1769076452.049157 punch from 0x01c553 to 0x02235b (0x5e09 bytes) 000280 1769076452.089687 MIRROR_RESYNC 4 mirrors 000281 1769076454.403580 mapread 0x013396 thru 0x016700 (0x336b bytes) 000282 1769076454.455179 read 0x014aee thru 0x01d72d (0x8c40 bytes) 000283 1769076454.456955 mapread 0x0367fe thru 0x03fb6d (0x9370 bytes) 000285 1769076454.482035 fallocate 0x00dd46 thru 0x01a875 (0xcb30 bytes) 000286 1769076454.533465 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x4666993a (stale) Mirror 8: 0x4666993a (stale) Mirror 9: 0x4666993a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4666993a Mirror 10: 0x4666993a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000288 1769076456.148766 fallocate 0x03d70a thru 0x03f0e2 (0x19d9 bytes) 000289 1769076456.183634 read 0x03bb94 thru 0x03ffff (0x446c bytes) 000290 1769076456.188911 read 0x006e14 thru 0x008da1 (0x1f8e bytes) 000291 1769076456.196520 trunc from 0x040000 to 0x033bc7 (0xc438 bytes) 000292 1769076456.214714 mapread 0x027c50 thru 0x02e090 (0x6441 bytes) 000293 1769076456.227679 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x33bc8): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33bc8): Mirror 7: 0xadd5b680 (stale) Mirror 8: 0xadd5b680 (stale) Mirror 9: 0xadd5b680 (stale) Mirror 10: 0xadd5b680 (stale) CRC-32 checksum value for chunk [0, 0x33bc8): Mirror 1: 0xadd5b680 Mirror 11: 0xadd5b680 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33bc8: skipped 000294 1769076457.378717 mapread 0x0093ad thru 0x00b999 (0x25ed bytes) 000295 1769076457.430474 trunc from 0x033bc8 to 0x03efda (0xb413 bytes) 000296 1769076457.512800 mapread 0x011c88 thru 0x017fed (0x6366 bytes) 000297 1769076457.523453 fallocate 0x032055 thru 0x037820 (0x57cc bytes) 000298 1769076457.528974 mapwrite 0x036c40 thru 0x03ffff (0x93c0 bytes) 000299 1769076457.619824 trunc from 0x0015f2 to 0x00bbc3 (0xa5d2 bytes) 000300 1769076457.627383 trunc from 0x040000 to 0x0353c2 (0xac3d bytes) 000301 1769076457.652363 MIRROR_RESYNC 6 mirrors 000303 1769076460.108850 mapwrite 0x017e53 thru 0x01eedd (0x708b bytes) 000304 1769076460.231684 fallocate 0x038f04 thru 0x03ffff (0x70fc bytes) 000305 1769076460.244541 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x353c3): Mirror 7: 0x961444d7 (stale) Mirror 8: 0x961444d7 (stale) Mirror 10: 0x961444d7 (stale) Mirror 11: 0x961444d7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x353c3): Mirror 7: 0x961444d7 (stale) Mirror 8: 0x961444d7 (stale) Mirror 10: 0x961444d7 (stale) Mirror 11: 0x961444d7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x353c3: skipped 000306 1769076460.750113 mapread 0x02d193 thru 0x03477d (0x75eb bytes) 000307 1769076460.771736 trunc from 0x0353c3 to 0x02544a (0xff78 bytes) 000308 1769076460.804815 write 0x003e2d thru 0x00ca7b (0x8c4f bytes) 000312 1769076460.818174 punch from 0x00b03e to 0x00e6dc (0x369f bytes) 000313 1769076460.833584 read 0x013489 thru 0x0218ba (0xe432 bytes) 000314 1769076460.847690 mapwrite 0x0245c3 thru 0x03102c (0xca6a bytes) 000315 1769076460.916013 mapwrite 0x01a9c4 thru 0x02298c (0x7fc9 bytes) 000316 1769076460.939676 write 0x015d2c thru 0x0205fd (0xa8d2 bytes) 000317 1769076460.946099 punch from 0x02d8cb to 0x030b2c (0x3262 bytes) 000318 1769076460.952632 mapread 0x02fd1e thru 0x03102c (0x130f bytes) 000319 1769076460.962931 punch from 0x02db06 to 0x03102c (0x3527 bytes) 000320 1769076460.971951 mapwrite 0x025f07 thru 0x0325eb (0xc6e5 bytes) 000321 1769076461.053091 trunc from 0x0325ec to 0x023185 (0xf466 bytes) 000323 1769076461.078804 trunc from 0x01e6da to 0x01eff3 (0x91a bytes) 000324 1769076461.087102 write 0x00ecec thru 0x01716a (0x847f bytes) 000325 1769076461.107808 mapread 0x01b6bf thru 0x02295a (0x729c bytes) 000326 1769076461.116807 mapwrite 0x034abd thru 0x03ffff (0xb543 bytes) 000327 1769076461.216524 mapwrite 0x01f94d thru 0x027a99 (0x814d bytes) 000328 1769076461.261115 fallocate 0x03132e thru 0x0337b6 (0x2489 bytes) 000329 1769076461.270397 trunc from 0x040000 to 0x009527 (0x36ad8 bytes) 000330 1769076461.315364 read 0x005937 thru 0x009527 (0x3bf1 bytes) 000331 1769076461.317384 read 0x003c45 thru 0x009527 (0x58e3 bytes) 000332 1769076461.320151 mapwrite 0x01133b thru 0x02031c (0xefe2 bytes) 000333 1769076461.399504 read 0x00a93d thru 0x019b2c (0xf1f0 bytes) 000334 1769076461.402081 fallocate 0x039e5d thru 0x03ffff (0x61a3 bytes) 000335 1769076461.409450 mapwrite 0x013e2e thru 0x019385 (0x5558 bytes) 000336 1769076461.427721 mapwrite 0x021f29 thru 0x0291af (0x7287 bytes) 000337 1769076461.517478 mapread 0x01bc08 thru 0x020fb9 (0x53b2 bytes) 000338 1769076461.522741 trunc from 0x0291b0 to 0x0397ef (0x10640 bytes) 000339 1769076461.588621 MIRROR_RESYNC 5 mirrors 000340 1769076463.561173 trunc from 0x00f19d to 0x00f5fa (0x45e bytes) 000341 1769076463.672663 mapwrite 0x012ce8 thru 0x021124 (0xe43d bytes) 000343 1769076463.733428 trunc from 0x0397f0 to 0x03853e (0x12b1 bytes) 000344 1769076463.774189 mapread 0x011836 thru 0x01b4fc (0x9cc7 bytes) 000345 1769076463.775318 punch from 0x001073 to 0x00b7d5 (0xa763 bytes) 000346 1769076463.793520 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3853f): Mirror 7: 0x8c3bfa5e (stale) Mirror 10: 0x8c3bfa5e (stale) Mirror 11: 0x8c3bfa5e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3853f): Mirror 7: 0x8c3bfa5e (stale) Mirror 10: 0x8c3bfa5e (stale) Mirror 11: 0x8c3bfa5e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3853f: skipped 000348 1769076464.359637 fallocate 0x00d323 thru 0x01b53f (0xe21d bytes) 000349 1769076464.367661 read 0x02aec1 thru 0x02b1a8 (0x2e8 bytes) 000350 1769076464.378665 trunc from 0x03595a to 0x03853e (0x2be5 bytes) 000351 1769076464.391928 trunc from 0x026a9f to 0x03671b (0xfc7d bytes) 000352 1769076464.402659 trunc from 0x0350fc to 0x03853e (0x3443 bytes) 000353 1769076464.408583 mapwrite 0x02c271 thru 0x02f913 (0x36a3 bytes) 000354 1769076464.493589 mapread 0x01d7f1 thru 0x01f3c1 (0x1bd1 bytes) 000355 1769076464.530393 read 0x01e5de thru 0x02e2c9 (0xfcec bytes) 000356 1769076464.566720 mapread 0x02da2e thru 0x03853e (0xab11 bytes) 000357 1769076464.572333 write 0x002a75 thru 0x00465e (0x1bea bytes) 000358 1769076464.583743 write 0x00c16c thru 0x012369 (0x61fe bytes) 000359 1769076464.586407 read 0x0155fd thru 0x0166e3 (0x10e7 bytes) 000361 1769076464.587987 trunc from 0x03853f to 0x02ad47 (0xd7f7 bytes) 000362 1769076464.643147 mapwrite 0x0067d8 thru 0x00f3b0 (0x8bd9 bytes) 000363 1769076464.672495 trunc from 0x02ad48 to 0x036edf (0xc198 bytes) 000364 1769076464.694454 read 0x0083ba thru 0x016b59 (0xe7a0 bytes) 000365 1769076464.695309 write 0x023eff thru 0x02cdef (0x8ef1 bytes) 000366 1769076464.704468 trunc from 0x026d58 to 0x02c14c (0x53f5 bytes) 000367 1769076464.730398 trunc from 0x01b5f0 to 0x027880 (0xc291 bytes) 000368 1769076464.741180 fallocate 0x01d8c5 thru 0x02724d (0x9989 bytes) 000369 1769076464.756855 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x36ee0): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ee0): Mirror 7: 0xe5c04959 (stale) Mirror 10: 0xe5c04959 (stale) Mirror 11: 0xe5c04959 (stale) CRC-32 checksum value for chunk [0, 0x36ee0): Mirror 1: 0x46cc47e8 Mirror 12: 0x46cc47e8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ee0: skipped 000370 1769076466.474160 trunc from 0x014a95 to 0x01f3bb (0xa927 bytes) 000371 1769076466.519802 write 0x02696c thru 0x028cb0 (0x2345 bytes) 000373 1769076466.543181 punch from 0x01f3d0 to 0x022281 (0x2eb2 bytes) 000374 1769076466.553813 fallocate 0x021540 thru 0x022475 (0xf36 bytes) 000375 1769076466.571501 trunc from 0x01468a to 0x01b163 (0x6ada bytes) 000376 1769076466.579727 trunc from 0x036ee0 to 0x00c74d (0x2a792 bytes) 000377 1769076466.634790 trunc from 0x00c74e to 0x03ae7c (0x2e72f bytes) 000378 1769076466.685989 write 0x00eaa7 thru 0x00f235 (0x78f bytes) 000379 1769076466.708516 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ae7d): Mirror 7: 0x556e2bd0 (stale) Mirror 10: 0x556e2bd0 (stale) Mirror 12: 0x43242fd7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ae7d): Mirror 7: 0x556e2bd0 (stale) Mirror 10: 0x556e2bd0 (stale) Mirror 12: 0x43242fd7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ae7d: skipped 000380 1769076467.273641 trunc from 0x0326ae to 0x03ae7c (0x87cf bytes) 000381 1769076467.286728 fallocate 0x006f4b thru 0x0086f0 (0x17a6 bytes) 000382 1769076467.298367 punch from 0x020ddb to 0x02c514 (0xb73a bytes) 000383 1769076467.307645 trunc from 0x011632 to 0x016ab8 (0x5487 bytes) 000384 1769076467.316673 fallocate 0x02054c thru 0x029355 (0x8e0a bytes) 000385 1769076467.324492 trunc from 0x03ae7d to 0x03b338 (0x4bc bytes) 000387 1769076467.370762 trunc from 0x03b339 to 0x021cf9 (0x1963f bytes) 000388 1769076467.405760 write 0x0006e9 thru 0x008cb6 (0x85ce bytes) 000389 1769076467.443166 trunc from 0x021cfa to 0x02a335 (0x863c bytes) 000391 1769076467.482865 fallocate 0x03a996 thru 0x03ffff (0x566a bytes) 000392 1769076467.495371 read 0x018dca thru 0x01d9c6 (0x4bfd bytes) 000393 1769076467.512730 mapread 0x013de7 thru 0x0204a1 (0xc6bb bytes) 000395 1769076467.541036 read 0x024740 thru 0x024ca4 (0x565 bytes) 000396 1769076467.541615 mapread 0x00f1d4 thru 0x01b30d (0xc13a bytes) 000397 1769076467.542480 mapwrite 0x038511 thru 0x03ffff (0x7aef bytes) 000399 1769076467.644501 fallocate 0x01f19a thru 0x028dcc (0x9c33 bytes) 000400 1769076467.664750 trunc from 0x040000 to 0x026a99 (0x19566 bytes) 000401 1769076467.720489 fallocate 0x01b8dd thru 0x024c13 (0x9337 bytes) 000402 1769076467.727116 read 0x001fb4 thru 0x00aca7 (0x8cf4 bytes) 000403 1769076467.727692 mapread 0x0040af thru 0x013f3c (0xfe8e bytes) 000404 1769076467.728637 trunc from 0x018aa5 to 0x0257d2 (0xcd2e bytes) 000407 1769076467.735541 fallocate 0x00f39b thru 0x014ff6 (0x5c5c bytes) 000408 1769076467.741335 mapwrite 0x030878 thru 0x035d60 (0x54e9 bytes) 000409 1769076467.813422 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x35d61): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35d61): Mirror 7: 0xddcfbcad (stale) Mirror 10: 0xddcfbcad (stale) Mirror 12: 0x818ffae3 (stale) CRC-32 checksum value for chunk [0, 0x35d61): Mirror 1: 0x45058e5 Mirror 13: 0x45058e5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35d61: skipped 000410 1769076468.728399 fallocate 0x02725b thru 0x02d52b (0x62d1 bytes) 000411 1769076468.774720 punch from 0x008335 to 0x00a29d (0x1f69 bytes) 000412 1769076468.781735 punch from 0x024b70 to 0x032564 (0xd9f5 bytes) 000413 1769076468.788062 trunc from 0x035d61 to 0x03d160 (0x7400 bytes) 000414 1769076468.813838 read 0x0391f7 thru 0x03d160 (0x3f6a bytes) 000415 1769076468.827688 mapread 0x017261 thru 0x0185b8 (0x1358 bytes) 000416 1769076468.837303 fallocate 0x006f57 thru 0x0130c8 (0xc172 bytes) 000418 1769076468.847846 MIRROR_RESYNC 5 mirrors 000419 1769076471.244042 fallocate 0x032928 thru 0x03f1b8 (0xc891 bytes) 000420 1769076471.298959 write 0x00d083 thru 0x015d7a (0x8cf8 bytes) 000421 1769076471.322339 read 0x0167d7 thru 0x01874c (0x1f76 bytes) 000422 1769076471.335105 mapread 0x02c41a thru 0x03b951 (0xf538 bytes) 000423 1769076471.366614 trunc from 0x03d161 to 0x00a1a7 (0x32fb9 bytes) 000424 1769076471.408160 trunc from 0x00064f to 0x00651d (0x5ecf bytes) 000426 1769076471.419147 write 0x032848 thru 0x032be9 (0x3a2 bytes) 000427 1769076471.421625 write 0x037aa4 thru 0x03ffff (0x855c bytes) 000428 1769076471.424243 trunc from 0x040000 to 0x02123c (0x1edc3 bytes) 000429 1769076471.471583 MIRROR_RESYNC 5 mirrors 000430 1769076473.601549 write 0x00c86f thru 0x00f590 (0x2d22 bytes) 000431 1769076473.642095 write 0x01576c thru 0x025644 (0xfed9 bytes) 000432 1769076473.653032 read 0x00be8d thru 0x019f7d (0xe0f1 bytes) 000433 1769076473.662024 mapwrite 0x018dbd thru 0x024f88 (0xc1cc bytes) 000434 1769076473.703470 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x25645): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25645): Mirror 7: 0xe927abfc (stale) Mirror 10: 0xe927abfc (stale) Mirror 12: 0xe927abfc (stale) Mirror 13: 0xe927abfc (stale) CRC-32 checksum value for chunk [0, 0x25645): Mirror 1: 0x4c633bd7 Mirror 14: 0x4c633bd7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25645: skipped 000435 1769076475.519754 write 0x01203f thru 0x01662c (0x45ee bytes) 000437 1769076475.584516 trunc from 0x025645 to 0x02d3c2 (0x7d7e bytes) 000440 1769076475.609545 trunc from 0x026112 to 0x02d3c2 (0x72b1 bytes) 000441 1769076475.615784 fallocate 0x026023 thru 0x02b663 (0x5641 bytes) 000442 1769076475.621411 trunc from 0x02d3c3 to 0x020b19 (0xc8a9 bytes) 000443 1769076475.646189 mapwrite 0x031260 thru 0x0315ab (0x34c bytes) 000444 1769076475.691673 punch from 0x0058bd to 0x00c3a2 (0x6ae6 bytes) 000445 1769076475.696840 read 0x01863e thru 0x021469 (0x8e2c bytes) 000446 1769076475.706248 mapwrite 0x01f54a thru 0x025f87 (0x6a3e bytes) 000447 1769076475.729397 fallocate 0x009483 thru 0x00ccd8 (0x3856 bytes) 000448 1769076475.734494 fallocate 0x02d0ea thru 0x0325b9 (0x54d0 bytes) 000449 1769076475.740184 read 0x0007ff thru 0x002efb (0x26fd bytes) 000450 1769076475.749343 fallocate 0x00b588 thru 0x01292f (0x73a8 bytes) 000451 1769076475.759519 MIRROR_RESYNC 6 mirrors 000452 1769076477.711658 trunc from 0x0325ba to 0x004cf4 (0x2d8c5 bytes) 000453 1769076477.763340 read 0x002943 thru 0x004cf4 (0x23b2 bytes) 000454 1769076477.771958 mapwrite 0x00d13b thru 0x016676 (0x953c bytes) 000455 1769076477.881338 mapread 0x00e4da thru 0x016676 (0x819d bytes) 000456 1769076477.882423 punch from 0x008ad8 to 0x00cffd (0x4526 bytes) 000458 1769076477.890560 trunc from 0x00109b to 0x00f8f9 (0xe85f bytes) 000459 1769076477.899388 fallocate 0x03e130 thru 0x03ffff (0x1ed0 bytes) 000460 1769076477.906204 mapwrite 0x02d570 thru 0x02e31a (0xdab bytes) 000461 1769076477.933664 trunc from 0x00b35d to 0x01404f (0x8cf3 bytes) 000462 1769076477.946643 MIRROR_RESYNC 6 mirrors 000463 1769076479.959742 mapread 0x01e3e6 thru 0x02879e (0xa3b9 bytes) 000464 1769076479.989038 write 0x036f87 thru 0x03ffff (0x9079 bytes) 000465 1769076480.068534 trunc from 0x040000 to 0x02d980 (0x1267f bytes) 000467 1769076480.123475 mapread 0x01777e thru 0x020e3d (0x96c0 bytes) 000468 1769076480.171971 mapwrite 0x0111d8 thru 0x01a46a (0x9293 bytes) 000469 1769076480.236816 punch from 0x01f3db to 0x02d980 (0xe5a6 bytes) 000470 1769076480.252659 trunc from 0x02d981 to 0x030927 (0x2fa7 bytes) 000471 1769076480.297536 mapread 0x00e4c8 thru 0x017f51 (0x9a8a bytes) 000472 1769076480.320542 trunc from 0x030928 to 0x0322dc (0x19b5 bytes) 000473 1769076480.348430 punch from 0x0042d8 to 0x010e34 (0xcb5d bytes) 000474 1769076480.366221 trunc from 0x00a5ff to 0x00abeb (0x5ed bytes) 000475 1769076480.387925 fallocate 0x01e28b thru 0x027f36 (0x9cac bytes) 000476 1769076480.409275 fallocate 0x01075f thru 0x01f397 (0xec39 bytes) 000477 1769076480.417560 read 0x01ba75 thru 0x01f12e (0x36ba bytes) 000478 1769076480.434373 trunc from 0x002bea to 0x01246e (0xf885 bytes) 000479 1769076480.455448 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x322dd): Mirror 7: 0xfb9d5b44 (stale) Mirror 10: 0xfb9d5b44 (stale) Mirror 12: 0xfb9d5b44 (stale) Mirror 13: 0xfb9d5b44 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x322dd): Mirror 7: 0xfb9d5b44 (stale) Mirror 10: 0xfb9d5b44 (stale) Mirror 12: 0xfb9d5b44 (stale) Mirror 13: 0xfb9d5b44 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x322dd: skipped 000480 1769076481.073307 read 0x01b3e9 thru 0x026db1 (0xb9c9 bytes) 000481 1769076481.090020 trunc from 0x0322dd to 0x02c397 (0x5f45 bytes) 000482 1769076481.128405 write 0x004567 thru 0x012a87 (0xe521 bytes) 000483 1769076481.149305 trunc from 0x02a066 to 0x02c397 (0x2332 bytes) 000484 1769076481.168786 mapread 0x0093a2 thru 0x010ee1 (0x7b40 bytes) 000485 1769076481.177796 fallocate 0x02a8a9 thru 0x036f3b (0xc693 bytes) 000487 1769076481.194101 punch from 0x01b54a to 0x024199 (0x8c50 bytes) 000488 1769076481.204340 mapread 0x0074f0 thru 0x012b2a (0xb63b bytes) 000489 1769076481.206440 trunc from 0x02b7c6 to 0x02f177 (0x39b2 bytes) 000490 1769076481.213150 trunc from 0x036f3c to 0x010e24 (0x26117 bytes) 000491 1769076481.240740 trunc from 0x010e25 to 0x035f9f (0x2517b bytes) 000492 1769076481.285558 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x35fa0): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35fa0): Mirror 7: 0x655786ba (stale) Mirror 10: 0x655786ba (stale) Mirror 12: 0x655786ba (stale) Mirror 13: 0x655786ba (stale) CRC-32 checksum value for chunk [0, 0x35fa0): Mirror 1: 0x30eac940 Mirror 14: 0x30eac940 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35fa0: skipped 000493 1769076482.578252 punch from 0x02f1e5 to 0x035f9f (0x6dbb bytes) 000494 1769076482.609480 punch from 0x01e07f to 0x023f60 (0x5ee2 bytes) 000495 1769076482.613337 mapwrite 0x026b1e thru 0x03446c (0xd94f bytes) 000496 1769076482.663374 read 0x0097dd thru 0x010b53 (0x7377 bytes) 000497 1769076482.672052 fallocate 0x02021c thru 0x027ea8 (0x7c8d bytes) 000498 1769076482.683995 punch from 0x00a0a0 to 0x00b990 (0x18f1 bytes) 000499 1769076482.697162 read 0x0303d2 thru 0x035f9f (0x5bce bytes) 000500 1769076482.698232 trunc from 0x002c9f to 0x0118f6 (0xec58 bytes) 000501 1769076482.710747 write 0x02ad10 thru 0x032287 (0x7578 bytes) 000502 1769076482.729590 mapread 0x001a52 thru 0x008588 (0x6b37 bytes) 000503 1769076482.750762 write 0x00f276 thru 0x01422f (0x4fba bytes) 000504 1769076482.752641 write 0x014720 thru 0x01958f (0x4e70 bytes) 000505 1769076482.754079 fallocate 0x028c6c thru 0x02b97f (0x2d14 bytes) 000507 1769076482.769297 punch from 0x0272e6 to 0x02f61f (0x833a bytes) 000509 1769076482.787270 trunc from 0x00562f to 0x010b51 (0xb523 bytes) 000510 1769076482.792319 mapread 0x035f09 thru 0x035f9f (0x97 bytes) 000511 1769076482.793186 punch from 0x004d45 to 0x0084fa (0x37b6 bytes) 000512 1769076482.798459 punch from 0x01768e to 0x01a215 (0x2b88 bytes) 000513 1769076482.803588 fallocate 0x035323 thru 0x03a1c9 (0x4ea7 bytes) 000514 1769076482.810990 write 0x00ac78 thru 0x01503b (0xa3c4 bytes) 000515 1769076482.820089 mapwrite 0x0399b2 thru 0x03f8d0 (0x5f1f bytes) 000516 1769076482.897336 mapwrite 0x002428 thru 0x00fb4e (0xd727 bytes) 000517 1769076482.957428 MIRROR_RESYNC 6 mirrors 000518 1769076485.049141 read 0x0270b8 thru 0x02d6bb (0x6604 bytes) 000520 1769076485.064606 trunc from 0x03f8d1 to 0x03ba7b (0x3e55 bytes) 000521 1769076485.120138 write 0x036bd2 thru 0x037509 (0x938 bytes) 000522 1769076485.132183 mapwrite 0x006a31 thru 0x009ea2 (0x3472 bytes) 000523 1769076485.171352 fallocate 0x01316d thru 0x022d4c (0xfbe0 bytes) 000524 1769076485.178686 mapread 0x00b24a thru 0x019265 (0xe01c bytes) 000525 1769076485.187434 mapread 0x03165c thru 0x03ba7b (0xa420 bytes) 000526 1769076485.205581 fallocate 0x0235a1 thru 0x032953 (0xf3b3 bytes) 000527 1769076485.213114 write 0x01c7c2 thru 0x0271df (0xaa1e bytes) 000528 1769076485.236513 trunc from 0x006138 to 0x00878d (0x2656 bytes) 000529 1769076485.241903 trunc from 0x01e746 to 0x02463e (0x5ef9 bytes) 000530 1769076485.260493 punch from 0x000b5e to 0x00b287 (0xa72a bytes) 000531 1769076485.265539 trunc from 0x03ba7c to 0x024af1 (0x16f8a bytes) 000532 1769076485.295798 fallocate 0x016e8f thru 0x024fc6 (0xe138 bytes) 000533 1769076485.303709 mapread 0x017e94 thru 0x01efe0 (0x714d bytes) 000534 1769076485.321301 trunc from 0x00550c to 0x00d325 (0x7e1a bytes) 000535 1769076485.336449 fallocate 0x02576a thru 0x033dbf (0xe656 bytes) 000536 1769076485.342462 read 0x0008ad thru 0x010365 (0xfab9 bytes) 000537 1769076485.351368 fallocate 0x0156a7 thru 0x02169e (0xbff8 bytes) 000538 1769076485.356623 read 0x0088ac thru 0x014e83 (0xc5d8 bytes) 000539 1769076485.357528 mapread 0x026eae thru 0x028cdf (0x1e32 bytes) 000540 1769076485.358498 read 0x01caa5 thru 0x026888 (0x9de4 bytes) 000541 1769076485.359487 read 0x02056f thru 0x028018 (0x7aaa bytes) 000542 1769076485.367563 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x33dc0): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33dc0): Mirror 7: 0xffefb3e9 (stale) Mirror 10: 0xffefb3e9 (stale) Mirror 12: 0xffefb3e9 (stale) Mirror 13: 0xffefb3e9 (stale) Mirror 14: 0xffefb3e9 (stale) CRC-32 checksum value for chunk [0, 0x33dc0): Mirror 1: 0x6c6bf59b Mirror 15: 0x6c6bf59b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33dc0: skipped 000543 1769076486.321789 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x33dc0): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33dc0): Mirror 7: 0xffefb3e9 (stale) Mirror 10: 0xffefb3e9 (stale) Mirror 12: 0xffefb3e9 (stale) Mirror 13: 0xffefb3e9 (stale) Mirror 14: 0xffefb3e9 (stale) CRC-32 checksum value for chunk [0, 0x33dc0): Mirror 1: 0x6c6bf59b Mirror 15: 0x6c6bf59b Mirror 16: 0x6c6bf59b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33dc0: skipped 000544 1769076487.342088 write 0x01826b thru 0x01ec4f (0x69e5 bytes) 000545 1769076487.397210 read 0x01ab82 thru 0x01c88f (0x1d0e bytes) 000547 1769076487.403479 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33dc0): Mirror 7: 0xffefb3e9 (stale) Mirror 10: 0xffefb3e9 (stale) Mirror 12: 0xffefb3e9 (stale) Mirror 13: 0xffefb3e9 (stale) Mirror 14: 0xffefb3e9 (stale) Mirror 15: 0x6c6bf59b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33dc0): Mirror 7: 0xffefb3e9 (stale) Mirror 10: 0xffefb3e9 (stale) Mirror 12: 0xffefb3e9 (stale) Mirror 13: 0xffefb3e9 (stale) Mirror 14: 0xffefb3e9 (stale) Mirror 15: 0x6c6bf59b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33dc0: skipped 000548 1769076487.895824 write 0x03f9bd thru 0x03ffff (0x643 bytes) 000549 1769076487.909366 fallocate 0x019b2b thru 0x0226a4 (0x8b7a bytes) 000550 1769076487.918560 trunc from 0x03388c to 0x0372fb (0x3a70 bytes) 000551 1769076487.928385 trunc from 0x040000 to 0x02dd5b (0x122a4 bytes) 000552 1769076487.981833 mapread 0x026a1c thru 0x02bd1b (0x5300 bytes) 000553 1769076488.004774 read 0x023656 thru 0x02dd5b (0xa706 bytes) 000554 1769076488.005719 punch from 0x0158cb to 0x01c4a2 (0x6bd8 bytes) 000555 1769076488.012345 trunc from 0x020e59 to 0x02dd5b (0xcf03 bytes) 000556 1769076488.018225 trunc from 0x02ac03 to 0x02dd5b (0x3159 bytes) 000557 1769076488.024724 fallocate 0x02bcb5 thru 0x037a86 (0xbdd2 bytes) 000558 1769076488.036349 MIRROR_RESYNC 7 mirrors 000559 1769076490.316237 read 0x0057d7 thru 0x00afae (0x57d8 bytes) 000560 1769076490.323723 punch from 0x0295f2 to 0x037a86 (0xe495 bytes) 000561 1769076490.367053 fallocate 0x02ed11 thru 0x03aba3 (0xbe93 bytes) 000562 1769076490.375623 trunc from 0x037a87 to 0x0004e8 (0x3759e bytes) 000563 1769076490.407301 fallocate 0x01ffe3 thru 0x0293ce (0x93ec bytes) 000564 1769076490.445718 fallocate 0x028833 thru 0x030f5e (0x872c bytes) 000566 1769076490.453456 fallocate 0x0241a4 thru 0x031dbc (0xdc19 bytes) 000567 1769076490.463370 fallocate 0x0117b4 thru 0x013fa0 (0x27ed bytes) 000568 1769076490.474978 punch from 0x01efcf to 0x02a1a1 (0xb1d3 bytes) 000569 1769076490.489566 trunc from 0x031dbd to 0x01996a (0x18452 bytes) 000570 1769076490.509917 read 0x006499 thru 0x012d16 (0xc87e bytes) 000571 1769076490.537696 fallocate 0x02800d thru 0x03237c (0xa370 bytes) 000572 1769076490.544732 read 0x00060d thru 0x00ae9f (0xa893 bytes) 000573 1769076490.545712 mapwrite 0x023c76 thru 0x02c79c (0x8b27 bytes) 000574 1769076490.615701 trunc from 0x02c79d to 0x00d178 (0x1f624 bytes) 000575 1769076490.647701 fallocate 0x024b7d thru 0x029e06 (0x528a bytes) 000576 1769076490.666671 write 0x006c59 thru 0x00fc67 (0x900f bytes) 000577 1769076490.671652 trunc from 0x00fc68 to 0x01fd4d (0x100e6 bytes) 000578 1769076490.717790 mapwrite 0x02f298 thru 0x03572f (0x6498 bytes) 000579 1769076490.774293 trunc from 0x02ca41 to 0x02dc18 (0x11d8 bytes) 000580 1769076490.794270 fallocate 0x017de8 thru 0x018185 (0x39e bytes) 000581 1769076490.815564 write 0x029900 thru 0x030ecb (0x75cc bytes) 000582 1769076490.832329 MIRROR_RESYNC 7 mirrors 000583 1769076492.896418 punch from 0x027c9c to 0x028c53 (0xfb8 bytes) 000584 1769076492.941638 write 0x036469 thru 0x03e4bd (0x8055 bytes) 000585 1769076492.944288 fallocate 0x00399f thru 0x007976 (0x3fd8 bytes) 000586 1769076492.950688 mapread 0x027f46 thru 0x02e5a1 (0x665c bytes) 000587 1769076492.971402 mapwrite 0x0180c3 thru 0x025a36 (0xd974 bytes) 000588 1769076493.025589 trunc from 0x03e4be to 0x00edf3 (0x2f6ca bytes) 000589 1769076493.060550 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xedf4): Mirror 10: 0x620b3979 (stale) Mirror 12: 0x620b3979 (stale) Mirror 13: 0x620b3979 (stale) Mirror 14: 0x620b3979 (stale) Mirror 15: 0x620b3979 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xedf4): Mirror 10: 0x620b3979 (stale) Mirror 12: 0x620b3979 (stale) Mirror 13: 0x620b3979 (stale) Mirror 14: 0x620b3979 (stale) Mirror 15: 0x620b3979 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xedf4: skipped 000590 1769076493.644042 trunc from 0x00edf4 to 0x03e68d (0x2f89a bytes) 000591 1769076493.681292 mapread 0x012026 thru 0x01309f (0x107a bytes) 000592 1769076493.696143 mapread 0x00c6eb thru 0x0114c3 (0x4dd9 bytes) 000593 1769076493.711597 MIRROR_RESYNC 6 mirrors 000594 1769076495.202437 trunc from 0x03632f to 0x03e68d (0x835f bytes) 000595 1769076495.298445 mapwrite 0x008891 thru 0x00a525 (0x1c95 bytes) 000597 1769076495.366514 trunc from 0x03e68e to 0x030ef0 (0xd79d bytes) 000598 1769076495.434055 punch from 0x0108c4 to 0x017816 (0x6f53 bytes) 000599 1769076495.453488 write 0x022d3d thru 0x028c4f (0x5f13 bytes) 000600 1769076495.475315 fallocate 0x0060bc thru 0x0153db (0xf320 bytes) 000601 1769076495.498144 MIRROR_RESYNC 6 mirrors 000602 1769076497.141753 mapwrite 0x017403 thru 0x01f17c (0x7d7a bytes) 000603 1769076497.298616 mapread 0x02ebf0 thru 0x030ef0 (0x2301 bytes) 000605 1769076497.311966 trunc from 0x025dcb to 0x030ef0 (0xb126 bytes) 000606 1769076497.318257 fallocate 0x0091ed thru 0x009cf0 (0xb04 bytes) 000607 1769076497.322814 punch from 0x014087 to 0x01f860 (0xb7da bytes) 000608 1769076497.330260 mapwrite 0x015c61 thru 0x01a255 (0x45f5 bytes) 000609 1769076497.363777 mapread 0x01289e thru 0x0169a8 (0x410b bytes) 000610 1769076497.364591 fallocate 0x03a5f1 thru 0x03ffff (0x5a0f bytes) 000612 1769076497.371817 fallocate 0x0002e0 thru 0x01021b (0xff3c bytes) 000613 1769076497.378116 trunc from 0x02bc04 to 0x02f05e (0x345b bytes) 000614 1769076497.385079 mapread 0x03af66 thru 0x03ffff (0x509a bytes) 000615 1769076497.395742 trunc from 0x02dd61 to 0x03c146 (0xe3e6 bytes) 000616 1769076497.402141 punch from 0x0248f8 to 0x0310e5 (0xc7ee bytes) 000617 1769076497.410752 fallocate 0x013caa thru 0x0160c1 (0x2418 bytes) 000618 1769076497.429679 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x1256ac39 (stale) Mirror 13: 0x1256ac39 (stale) Mirror 14: 0x1256ac39 (stale) Mirror 15: 0x1256ac39 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x1256ac39 (stale) Mirror 13: 0x1256ac39 (stale) Mirror 14: 0x1256ac39 (stale) Mirror 15: 0x1256ac39 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000619 1769076497.872504 trunc from 0x040000 to 0x03f0c7 (0xf38 bytes) 000620 1769076497.899675 punch from 0x0356a2 to 0x03daff (0x845e bytes) 000621 1769076497.910700 fallocate 0x02e443 thru 0x034a64 (0x6622 bytes) 000622 1769076497.918787 mapwrite 0x01958e thru 0x01dd20 (0x4793 bytes) 000623 1769076497.964507 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3f0c8): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f0c8): Mirror 12: 0x434ba0b3 (stale) Mirror 13: 0x434ba0b3 (stale) Mirror 14: 0x434ba0b3 (stale) Mirror 15: 0x434ba0b3 (stale) CRC-32 checksum value for chunk [0, 0x3f0c8): Mirror 1: 0x510bdd26 Mirror 16: 0x510bdd26 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f0c8: skipped 000624 1769076499.034642 mapread 0x01be9f thru 0x024e43 (0x8fa5 bytes) 000625 1769076499.107609 read 0x017714 thru 0x01a41a (0x2d07 bytes) 000626 1769076499.119334 write 0x0355ab thru 0x038cd9 (0x372f bytes) 000627 1769076499.220602 mapwrite 0x0315ba thru 0x0322c0 (0xd07 bytes) 000628 1769076499.248253 read 0x0310fa thru 0x039fbc (0x8ec3 bytes) 000629 1769076499.248844 mapwrite 0x037808 thru 0x037e9e (0x697 bytes) 000631 1769076499.278619 trunc from 0x03f0c8 to 0x022b95 (0x1c532 bytes) 000632 1769076499.304260 read 0x0166e2 thru 0x022801 (0xc120 bytes) 000634 1769076499.324098 mapwrite 0x018e0d thru 0x025a35 (0xcc29 bytes) 000635 1769076499.378854 fallocate 0x00803f thru 0x00a481 (0x2443 bytes) 000636 1769076499.389605 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x25a36): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25a36): Mirror 12: 0xce6bf7af (stale) Mirror 13: 0xce6bf7af (stale) Mirror 14: 0xce6bf7af (stale) Mirror 15: 0xce6bf7af (stale) Mirror 16: 0x6354433e (stale) CRC-32 checksum value for chunk [0, 0x25a36): Mirror 1: 0x7f05950e Mirror 17: 0x7f05950e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25a36: skipped 000637 1769076500.317549 MIRROR_RESYNC 7 mirrors 000638 1769076501.523360 fallocate 0x02ad3c thru 0x02d065 (0x232a bytes) 000639 1769076501.591468 write 0x00c7d3 thru 0x01a96c (0xe19a bytes) 000640 1769076501.608187 punch from 0x01c09e to 0x028925 (0xc888 bytes) 000641 1769076501.614050 trunc from 0x02d066 to 0x030d95 (0x3d30 bytes) 000642 1769076501.657598 trunc from 0x030d96 to 0x0202d3 (0x10ac2 bytes) 000643 1769076501.676619 trunc from 0x0176bc to 0x0202d3 (0x8c18 bytes) 000644 1769076501.691478 trunc from 0x0093c4 to 0x0144bf (0xb0fc bytes) 000645 1769076501.702177 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x202d4): Mirror 12: 0xd0fadc1e (stale) Mirror 13: 0xd0fadc1e (stale) Mirror 14: 0xd0fadc1e (stale) Mirror 16: 0xd0fadc1e (stale) Mirror 17: 0xd0fadc1e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x202d4): Mirror 12: 0xd0fadc1e (stale) Mirror 13: 0xd0fadc1e (stale) Mirror 14: 0xd0fadc1e (stale) Mirror 16: 0xd0fadc1e (stale) Mirror 17: 0xd0fadc1e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x202d4: skipped 000646 1769076502.256810 fallocate 0x01a5de thru 0x01b0c3 (0xae6 bytes) 000647 1769076502.269860 punch from 0x01cae0 to 0x0202d3 (0x37f4 bytes) 000648 1769076502.286204 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x202d4): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x202d4): Mirror 12: 0xd0fadc1e (stale) Mirror 13: 0xd0fadc1e (stale) Mirror 14: 0xd0fadc1e (stale) Mirror 16: 0xd0fadc1e (stale) Mirror 17: 0xd0fadc1e (stale) CRC-32 checksum value for chunk [0, 0x202d4): Mirror 1: 0xf333d49 Mirror 18: 0xf333d49 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x202d4: skipped 000650 1769076503.534855 MIRROR_RESYNC 7 mirrors 000651 1769076505.223648 fallocate 0x001d4f thru 0x006a73 (0x4d25 bytes) 000652 1769076505.272916 fallocate 0x03beb6 thru 0x03ff60 (0x40ab bytes) 000653 1769076505.279610 trunc from 0x039d86 to 0x03ba74 (0x1cef bytes) 000654 1769076505.284701 read 0x034001 thru 0x03ff60 (0xbf60 bytes) 000655 1769076505.295189 trunc from 0x0320cf to 0x032375 (0x2a7 bytes) 000656 1769076505.300073 fallocate 0x0113fd thru 0x01b683 (0xa287 bytes) 000657 1769076505.306562 trunc from 0x03ff61 to 0x02234a (0x1dc16 bytes) 000658 1769076505.333699 fallocate 0x02a84d thru 0x02d1f8 (0x29ac bytes) 000659 1769076505.343035 trunc from 0x02d1f9 to 0x03b352 (0xe15a bytes) 000660 1769076505.357386 fallocate 0x00a1ca thru 0x00c7d6 (0x260d bytes) 000661 1769076505.361663 write 0x038fb2 thru 0x03f009 (0x6058 bytes) 000662 1769076505.367067 punch from 0x0287b1 to 0x02d415 (0x4c65 bytes) 000664 1769076505.373809 mapwrite 0x021d51 thru 0x029bc4 (0x7e74 bytes) 000665 1769076505.404200 trunc from 0x03f00a to 0x00d981 (0x31688 bytes) 000667 1769076505.419721 mapread 0x00733f thru 0x00d981 (0x6643 bytes) 000668 1769076505.425488 trunc from 0x00269b to 0x00d981 (0xb2e7 bytes) 000669 1769076505.428667 trunc from 0x00d982 to 0x0376ca (0x29d49 bytes) 000670 1769076505.449383 mapwrite 0x007c49 thru 0x00f0d9 (0x7491 bytes) 000671 1769076505.480396 punch from 0x010a9e to 0x010e37 (0x39a bytes) 000673 1769076505.486689 mapwrite 0x026d09 thru 0x03331b (0xc613 bytes) 000674 1769076505.522661 write 0x035930 thru 0x03a3b1 (0x4a82 bytes) 000675 1769076505.528856 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a3b2): Mirror 13: 0x4a3f8d79 (stale) Mirror 14: 0x4a3f8d79 (stale) Mirror 16: 0x4a3f8d79 (stale) Mirror 17: 0x4a3f8d79 (stale) Mirror 18: 0x4a3f8d79 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a3b2): Mirror 13: 0x4a3f8d79 (stale) Mirror 14: 0x4a3f8d79 (stale) Mirror 16: 0x4a3f8d79 (stale) Mirror 17: 0x4a3f8d79 (stale) Mirror 18: 0x4a3f8d79 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a3b2: skipped 000676 1769076506.047308 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a3b2): Mirror 13: 0x4a3f8d79 (stale) Mirror 14: 0x4a3f8d79 (stale) Mirror 16: 0x4a3f8d79 (stale) Mirror 17: 0x4a3f8d79 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a3b2): Mirror 13: 0x4a3f8d79 (stale) Mirror 14: 0x4a3f8d79 (stale) Mirror 16: 0x4a3f8d79 (stale) Mirror 17: 0x4a3f8d79 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a3b2: skipped 000677 1769076506.554384 mapread 0x01dcf8 thru 0x028d9e (0xb0a7 bytes) 000678 1769076506.627804 punch from 0x02b725 to 0x036776 (0xb052 bytes) 000679 1769076506.642794 write 0x02ba44 thru 0x0326da (0x6c97 bytes) 000680 1769076506.695266 MIRROR_RESYNC 5 mirrors 000681 1769076508.572539 punch from 0x031a50 to 0x03a3b1 (0x8962 bytes) 000682 1769076508.630203 mapread 0x014f41 thru 0x0164ba (0x157a bytes) 000683 1769076508.662281 write 0x035939 thru 0x03ffff (0xa6c7 bytes) 000684 1769076508.690374 MIRROR_RESYNC 5 mirrors 000685 1769076510.840422 mapwrite 0x016c42 thru 0x01fa4f (0x8e0e bytes) 000686 1769076510.972785 write 0x03e164 thru 0x03ffff (0x1e9c bytes) 000687 1769076510.988429 punch from 0x014c57 to 0x015ffb (0x13a5 bytes) 000688 1769076511.005625 trunc from 0x019126 to 0x024f63 (0xbe3e bytes) 000690 1769076511.012217 trunc from 0x01a61f to 0x028ea8 (0xe88a bytes) 000691 1769076511.029098 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x21e01830 (stale) Mirror 14: 0x21e01830 (stale) Mirror 16: 0x21e01830 (stale) Mirror 17: 0x21e01830 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x99d54336 Mirror 18: 0x99d54336 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000693 1769076512.226324 trunc from 0x040000 to 0x0121a1 (0x2de5e bytes) 000694 1769076512.289612 trunc from 0x010940 to 0x010ba9 (0x26a bytes) 000696 1769076512.295184 punch from 0x011814 to 0x0121a1 (0x98e bytes) 000697 1769076512.300247 fallocate 0x008100 thru 0x013e09 (0xbd0a bytes) 000698 1769076512.305002 mapread 0x00ac56 thru 0x0121a1 (0x754c bytes) 000699 1769076512.331274 trunc from 0x011ced to 0x0121a1 (0x4b5 bytes) 000700 1769076512.337175 write 0x00a1e7 thru 0x014ed4 (0xacee bytes) 000701 1769076512.340373 read 0x0106a0 thru 0x014431 (0x3d92 bytes) 000702 1769076512.341405 write 0x0135e1 thru 0x01c0e4 (0x8b04 bytes) 000703 1769076512.342814 punch from 0x00face to 0x01189f (0x1dd2 bytes) 000704 1769076512.368629 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c0e5): Mirror 14: 0xb552f3d5 (stale) Mirror 16: 0xb552f3d5 (stale) Mirror 17: 0xb552f3d5 (stale) Mirror 18: 0xa58d2730 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c0e5): Mirror 14: 0xb552f3d5 (stale) Mirror 16: 0xb552f3d5 (stale) Mirror 17: 0xb552f3d5 (stale) Mirror 18: 0xa58d2730 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c0e5: skipped 000705 1769076512.753690 mapread 0x01bf31 thru 0x01c0e4 (0x1b4 bytes) 000706 1769076512.769415 mapread 0x002200 thru 0x008a29 (0x682a bytes) 000707 1769076512.780495 read 0x00b96b thru 0x00fe36 (0x44cc bytes) 000708 1769076512.781374 trunc from 0x01c0e5 to 0x03ebaf (0x22acb bytes) 000709 1769076512.811249 MIRROR_RESYNC 5 mirrors 000710 1769076514.692288 mapwrite 0x03bdcf thru 0x03ffff (0x4231 bytes) 000711 1769076514.807973 trunc from 0x040000 to 0x02895a (0x176a5 bytes) 000712 1769076514.840123 mapread 0x01436f thru 0x01e3e7 (0xa079 bytes) 000713 1769076514.870391 mapread 0x01f345 thru 0x02895a (0x9616 bytes) 000714 1769076514.881094 read 0x011ed8 thru 0x01c3f7 (0xa520 bytes) 000715 1769076514.882923 trunc from 0x02895b to 0x026e48 (0x1b12 bytes) 000716 1769076514.913186 trunc from 0x026e49 to 0x0131eb (0x13c5d bytes) 000717 1769076514.943040 trunc from 0x0131ec to 0x0061b8 (0xd033 bytes) 000718 1769076514.965773 read 0x000869 thru 0x002e33 (0x25cb bytes) 000720 1769076514.979003 trunc from 0x0061b9 to 0x012c7e (0xcac6 bytes) 000721 1769076515.001252 write 0x0397a4 thru 0x03a8dd (0x113a bytes) 000722 1769076515.002264 trunc from 0x03a8de to 0x0313c4 (0x9519 bytes) 000723 1769076515.025217 fallocate 0x00019b thru 0x005d8d (0x5bf3 bytes) 000724 1769076515.033210 trunc from 0x0313c5 to 0x019995 (0x17a2f bytes) 000725 1769076515.057435 write 0x012fd0 thru 0x01b6aa (0x86db bytes) 000726 1769076515.072564 fallocate 0x02a292 thru 0x02c9cd (0x273c bytes) 000727 1769076515.079074 trunc from 0x01b6ab to 0x01a39d (0x130d bytes) 000728 1769076515.105493 mapwrite 0x039600 thru 0x03ffff (0x6a00 bytes) 000729 1769076515.161380 read 0x03f9f9 thru 0x03ffff (0x607 bytes) 000730 1769076515.163276 mapwrite 0x033e9a thru 0x03ff6d (0xc0d4 bytes) 000731 1769076515.188999 trunc from 0x040000 to 0x01bb70 (0x2448f bytes) 000732 1769076515.214098 trunc from 0x01397b to 0x01bb70 (0x81f6 bytes) 000733 1769076515.222315 fallocate 0x03f200 thru 0x03ffff (0xe00 bytes) 000734 1769076515.230084 mapwrite 0x02586e thru 0x033ee1 (0xe674 bytes) 000735 1769076515.296711 trunc from 0x014a64 to 0x01c640 (0x7bdd bytes) 000736 1769076515.305588 punch from 0x00a267 to 0x0147c1 (0xa55b bytes) 000737 1769076515.311235 mapread 0x0136ac thru 0x01859d (0x4ef2 bytes) 000738 1769076515.349379 fallocate 0x0016be thru 0x002bb5 (0x14f8 bytes) 000739 1769076515.356457 punch from 0x02f620 to 0x033ee1 (0x48c2 bytes) 000740 1769076515.365992 mapwrite 0x038e2e thru 0x03e273 (0x5446 bytes) 000741 1769076515.479206 mapwrite 0x0146dd thru 0x01cbc3 (0x84e7 bytes) 000742 1769076515.623643 read 0x02b06c thru 0x037e50 (0xcde5 bytes) 000743 1769076515.649926 trunc from 0x03e274 to 0x0377d1 (0x6aa2 bytes) 000744 1769076515.683354 trunc from 0x0088e2 to 0x012006 (0x9725 bytes) 000746 1769076515.691864 trunc from 0x0377d2 to 0x00b390 (0x2c441 bytes) 000747 1769076515.742372 punch from 0x00027f to 0x001fc9 (0x1d4b bytes) 000748 1769076515.751119 mapwrite 0x0009b2 thru 0x00dc4f (0xd29e bytes) 000749 1769076515.916617 mapwrite 0x03bf11 thru 0x03ffff (0x40ef bytes) 000750 1769076516.071379 trunc from 0x040000 to 0x01ce8c (0x23173 bytes) 000751 1769076516.094164 mapwrite 0x03860a thru 0x03ffff (0x79f6 bytes) 000753 1769076516.235242 mapwrite 0x01ddef thru 0x027efb (0xa10d bytes) 000754 1769076516.352616 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xe667f1e7 (stale) Mirror 16: 0xe667f1e7 (stale) Mirror 17: 0xe667f1e7 (stale) Mirror 18: 0xe667f1e7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x34d840e8 Mirror 19: 0x34d840e8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000757 1769076517.493309 mapread 0x003c6a thru 0x009c93 (0x602a bytes) 000758 1769076517.546280 write 0x0166a3 thru 0x024e44 (0xe7a2 bytes) 000759 1769076517.625407 punch from 0x00c5cd to 0x01bc99 (0xf6cd bytes) 000760 1769076517.649495 trunc from 0x040000 to 0x01752b (0x28ad4 bytes) 000761 1769076517.720054 mapread 0x000df1 thru 0x00933e (0x854e bytes) 000762 1769076517.735353 trunc from 0x00c8d6 to 0x01752b (0xac56 bytes) 000765 1769076517.748390 MIRROR_RESYNC 6 mirrors 000768 1769076518.948250 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) 000769 1769076519.344694 mapread 0x00940b thru 0x00d8b3 (0x44a9 bytes) 000770 1769076519.385465 write 0x02f629 thru 0x03f464 (0xfe3c bytes) 000771 1769076519.462652 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f465): Mirror 16: 0x4aeb79d (stale) Mirror 17: 0x4aeb79d (stale) Mirror 19: 0x4aeb79d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f465): Mirror 16: 0x4aeb79d (stale) Mirror 17: 0x4aeb79d (stale) Mirror 19: 0x4aeb79d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f465: skipped 000772 1769076519.798869 trunc from 0x03f465 to 0x030138 (0xf32c bytes) 000773 1769076519.822644 mapwrite 0x0386fc thru 0x03ffff (0x7904 bytes) 000774 1769076519.899328 trunc from 0x040000 to 0x007438 (0x38bc7 bytes) 000775 1769076519.924599 MIRROR_RESYNC 4 mirrors 000776 1769076520.524730 MIRROR_EXTEND to 5 mirrors 000777 1769076521.005805 trunc from 0x007439 to 0x009d59 (0x2921 bytes) 000778 1769076521.087598 read 0x000cf1 thru 0x009d59 (0x9069 bytes) 000779 1769076521.135615 MIRROR_RESYNC 5 mirrors 000780 1769076522.179600 trunc from 0x009d5a to 0x02073a (0x169e1 bytes) 000781 1769076522.247962 mapwrite 0x031d5d thru 0x03d588 (0xb82c bytes) 000782 1769076522.321025 trunc from 0x02ab2f to 0x02d401 (0x28d3 bytes) 000783 1769076522.325623 fallocate 0x020421 thru 0x02dabb (0xd69b bytes) 000784 1769076522.333180 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3d589): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d589): Mirror 16: 0xa684bfec (stale) Mirror 17: 0xa684bfec (stale) Mirror 19: 0xa684bfec (stale) Mirror 20: 0xa684bfec (stale) CRC-32 checksum value for chunk [0, 0x3d589): Mirror 1: 0xfae4a956 Mirror 21: 0xfae4a956 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d589: skipped 000785 1769076523.068452 trunc from 0x03d589 to 0x0021e4 (0x3b3a4 bytes) 000786 1769076523.168597 trunc from 0x00194c to 0x0021e4 (0x899 bytes) 000787 1769076523.179534 MIRROR_RESYNC 6 mirrors 000789 1769076523.845031 write 0x01a9fd thru 0x0241c2 (0x97c6 bytes) 000790 1769076523.883450 read 0x01d4b6 thru 0x0241c2 (0x6d0d bytes) 000792 1769076523.892210 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x241c3): Mirror 16: 0x40d0a0bf (stale) Mirror 17: 0x40d0a0bf (stale) Mirror 19: 0x40d0a0bf (stale) Mirror 21: 0x40d0a0bf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x241c3): Mirror 16: 0x40d0a0bf (stale) Mirror 17: 0x40d0a0bf (stale) Mirror 19: 0x40d0a0bf (stale) Mirror 21: 0x40d0a0bf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x241c3: skipped 000793 1769076524.286156 MIRROR_RESYNC 5 mirrors 000795 1769076525.085587 fallocate 0x0273de thru 0x02ca05 (0x5628 bytes) 000796 1769076525.124668 read 0x01e7da thru 0x0241c2 (0x59e9 bytes) 000797 1769076525.140367 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x241c3): Mirror 16: 0xfb59abe1 (stale) Mirror 17: 0xfb59abe1 (stale) Mirror 19: 0xfb59abe1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x241c3): Mirror 16: 0xfb59abe1 (stale) Mirror 17: 0xfb59abe1 (stale) Mirror 19: 0xfb59abe1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x241c3: skipped 000798 1769076525.538512 trunc from 0x023a14 to 0x0241c2 (0x7af bytes) 000799 1769076525.545291 trunc from 0x0165ea to 0x0171a5 (0xbbc bytes) 000800 1769076525.556817 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x241c3): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x241c3): Mirror 16: 0xfb59abe1 (stale) Mirror 17: 0xfb59abe1 (stale) Mirror 19: 0xfb59abe1 (stale) CRC-32 checksum value for chunk [0, 0x241c3): Mirror 1: 0x85087168 Mirror 20: 0x85087168 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x241c3: skipped 000801 1769076526.329323 trunc from 0x012f20 to 0x0164b9 (0x359a bytes) 000802 1769076526.373660 mapread 0x000043 thru 0x002132 (0x20f0 bytes) 000803 1769076526.393396 mapread 0x00da3a thru 0x018b8e (0xb155 bytes) 000804 1769076526.415422 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x241c3): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x241c3): Mirror 16: 0xfb59abe1 (stale) Mirror 17: 0xfb59abe1 (stale) Mirror 19: 0xfb59abe1 (stale) Mirror 20: 0x85087168 (stale) CRC-32 checksum value for chunk [0, 0x241c3): Mirror 1: 0x85087168 Mirror 21: 0x85087168 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x241c3: skipped 000805 1769076527.179648 mapread 0x010174 thru 0x01dc18 (0xdaa5 bytes) 000806 1769076527.227229 mapwrite 0x0020ac thru 0x00ea23 (0xc978 bytes) 000808 1769076527.354400 read 0x010594 thru 0x013263 (0x2cd0 bytes) 000809 1769076527.366489 punch from 0x002890 to 0x0084ca (0x5c3b bytes) 000810 1769076527.383383 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x241c3): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x241c3): Mirror 16: 0xfb59abe1 (stale) Mirror 17: 0xfb59abe1 (stale) Mirror 19: 0xfb59abe1 (stale) Mirror 20: 0x85087168 (stale) Mirror 21: 0x85087168 (stale) CRC-32 checksum value for chunk [0, 0x241c3): Mirror 1: 0xd1e87e24 Mirror 22: 0xd1e87e24 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x241c3: skipped 000811 1769076528.582361 read 0x0106dd thru 0x019b20 (0x9444 bytes) 000812 1769076528.600120 mapread 0x023a54 thru 0x0241c2 (0x76f bytes) 000813 1769076528.631777 fallocate 0x03d8cb thru 0x03ffff (0x2735 bytes) 000814 1769076528.688421 mapwrite 0x03568e thru 0x03ffff (0xa972 bytes) 000815 1769076528.746209 mapread 0x022246 thru 0x022250 (0xb bytes) 000816 1769076528.759577 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xe43c503e (stale) Mirror 17: 0xe43c503e (stale) Mirror 19: 0xe43c503e (stale) Mirror 20: 0x89cb87fe (stale) Mirror 21: 0x89cb87fe (stale) Mirror 22: 0xf77ace7b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x69bc7a30 Mirror 23: 0x69bc7a30 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000817 1769076529.744721 mapread 0x008c53 thru 0x0123f8 (0x97a6 bytes) 000819 1769076529.836942 punch from 0x038cf1 to 0x03c03f (0x334f bytes) 000820 1769076529.933376 mapwrite 0x01e93a thru 0x02b4a9 (0xcb70 bytes) 000821 1769076530.016821 write 0x0103da thru 0x016e12 (0x6a39 bytes) 000822 1769076530.034332 trunc from 0x040000 to 0x033c61 (0xc39e bytes) 000823 1769076530.082292 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x33c62): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33c62): Mirror 16: 0x40803a73 (stale) Mirror 17: 0x40803a73 (stale) Mirror 19: 0x40803a73 (stale) Mirror 20: 0xcf603093 (stale) Mirror 21: 0xcf603093 (stale) Mirror 22: 0x3b9dfaab (stale) Mirror 23: 0x3b9dfaab (stale) CRC-32 checksum value for chunk [0, 0x33c62): Mirror 1: 0x96b8ac75 Mirror 24: 0x96b8ac75 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33c62: skipped 000824 1769076531.098030 fallocate 0x030c36 thru 0x031c21 (0xfec bytes) 000825 1769076531.154233 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x33c62): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33c62): Mirror 16: 0x40803a73 (stale) Mirror 17: 0x40803a73 (stale) Mirror 19: 0x40803a73 (stale) Mirror 20: 0xcf603093 (stale) Mirror 21: 0xcf603093 (stale) Mirror 22: 0x3b9dfaab (stale) Mirror 23: 0x3b9dfaab (stale) Mirror 24: 0x96b8ac75 (stale) CRC-32 checksum value for chunk [0, 0x33c62): Mirror 1: 0x96b8ac75 Mirror 25: 0x96b8ac75 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33c62: skipped 000826 1769076532.386313 punch from 0x020fea to 0x02e7bd (0xd7d4 bytes) 000827 1769076532.474690 read 0x01cabf thru 0x0253c0 (0x8902 bytes) 000829 1769076532.484124 trunc from 0x009a67 to 0x017ae8 (0xe082 bytes) 000830 1769076532.490827 write 0x0380c9 thru 0x03ffff (0x7f37 bytes) 000832 1769076532.501187 write 0x013154 thru 0x017cb5 (0x4b62 bytes) 000833 1769076532.510763 trunc from 0x031e01 to 0x032792 (0x992 bytes) 000834 1769076532.534563 MIRROR_RESYNC 10 mirrors 000835 1769076536.700911 trunc from 0x040000 to 0x00046a (0x3fb95 bytes) 000836 1769076536.748055 trunc from 0x0000ba to 0x00046a (0x3b1 bytes) 000837 1769076536.753605 mapwrite 0x030870 thru 0x03ecb5 (0xe446 bytes) 000838 1769076536.816305 write 0x00d316 thru 0x0182fe (0xafe9 bytes) 000839 1769076536.834465 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3ecb6): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ecb6): Mirror 16: 0xa991e7ee (stale) Mirror 17: 0xa991e7ee (stale) Mirror 19: 0xa991e7ee (stale) Mirror 20: 0xa991e7ee (stale) Mirror 21: 0xa991e7ee (stale) Mirror 22: 0xa991e7ee (stale) Mirror 23: 0xa991e7ee (stale) Mirror 24: 0xa991e7ee (stale) Mirror 25: 0xa991e7ee (stale) CRC-32 checksum value for chunk [0, 0x3ecb6): Mirror 1: 0xbc6ab97e Mirror 26: 0xbc6ab97e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ecb6: skipped 000840 1769076538.823418 punch from 0x004032 to 0x00c1af (0x817e bytes) 000841 1769076538.899433 mapread 0x0249a7 thru 0x032477 (0xdad1 bytes) 000842 1769076538.948859 write 0x0382de thru 0x03e4b7 (0x61da bytes) 000843 1769076538.977235 MIRROR_RESYNC 11 mirrors 000844 1769076542.203011 write 0x002dd0 thru 0x005d4b (0x2f7c bytes) 000845 1769076542.327823 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ecb6): Mirror 16: 0x699187b9 (stale) Mirror 17: 0x699187b9 (stale) Mirror 19: 0x699187b9 (stale) Mirror 21: 0x699187b9 (stale) Mirror 22: 0x699187b9 (stale) Mirror 23: 0x699187b9 (stale) Mirror 24: 0x699187b9 (stale) Mirror 25: 0x699187b9 (stale) Mirror 26: 0x699187b9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ecb6): Mirror 16: 0x699187b9 (stale) Mirror 17: 0x699187b9 (stale) Mirror 19: 0x699187b9 (stale) Mirror 21: 0x699187b9 (stale) Mirror 22: 0x699187b9 (stale) Mirror 23: 0x699187b9 (stale) Mirror 24: 0x699187b9 (stale) Mirror 25: 0x699187b9 (stale) Mirror 26: 0x699187b9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ecb6: skipped 000847 1769076543.146459 trunc from 0x012374 to 0x013b80 (0x180d bytes) 000848 1769076543.162186 mapread 0x01e995 thru 0x02c4cd (0xdb39 bytes) 000849 1769076543.195789 trunc from 0x03ecb6 to 0x023f15 (0x1ada0 bytes) 000850 1769076543.225603 trunc from 0x023f16 to 0x01f932 (0x45e3 bytes) 000851 1769076543.257794 read 0x004ca8 thru 0x0067e6 (0x1b3f bytes) 000852 1769076543.272171 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x1f933): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f933): Mirror 16: 0xe15e989 (stale) Mirror 17: 0xe15e989 (stale) Mirror 19: 0xe15e989 (stale) Mirror 21: 0xe15e989 (stale) Mirror 22: 0xe15e989 (stale) Mirror 23: 0xe15e989 (stale) Mirror 24: 0xe15e989 (stale) Mirror 25: 0xe15e989 (stale) Mirror 26: 0xe15e989 (stale) CRC-32 checksum value for chunk [0, 0x1f933): Mirror 1: 0x950dd321 Mirror 27: 0x950dd321 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f933: skipped 000853 1769076544.887017 trunc from 0x007411 to 0x01672a (0xf31a bytes) 000854 1769076544.963639 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f933): Mirror 16: 0xe15e989 (stale) Mirror 17: 0xe15e989 (stale) Mirror 19: 0xe15e989 (stale) Mirror 21: 0xe15e989 (stale) Mirror 23: 0xe15e989 (stale) Mirror 24: 0xe15e989 (stale) Mirror 25: 0xe15e989 (stale) Mirror 26: 0xe15e989 (stale) Mirror 27: 0x950dd321 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f933): Mirror 16: 0xe15e989 (stale) Mirror 17: 0xe15e989 (stale) Mirror 19: 0xe15e989 (stale) Mirror 21: 0xe15e989 (stale) Mirror 23: 0xe15e989 (stale) Mirror 24: 0xe15e989 (stale) Mirror 25: 0xe15e989 (stale) Mirror 26: 0xe15e989 (stale) Mirror 27: 0x950dd321 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f933: skipped 000856 1769076545.698722 fallocate 0x013d14 thru 0x01db04 (0x9df1 bytes) 000857 1769076545.713441 fallocate 0x013515 thru 0x01ad5e (0x784a bytes) 000858 1769076545.720463 write 0x02be34 thru 0x038266 (0xc433 bytes) 000859 1769076545.734213 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x38267): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38267): Mirror 16: 0x5bf6edd9 (stale) Mirror 17: 0x5bf6edd9 (stale) Mirror 19: 0x5bf6edd9 (stale) Mirror 21: 0x5bf6edd9 (stale) Mirror 23: 0x5bf6edd9 (stale) Mirror 24: 0x5bf6edd9 (stale) Mirror 25: 0x5bf6edd9 (stale) Mirror 26: 0x5bf6edd9 (stale) Mirror 27: 0x7ea1b63d (stale) CRC-32 checksum value for chunk [0, 0x38267): Mirror 1: 0xa4cca4bd Mirror 28: 0xa4cca4bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38267: skipped 000860 1769076546.920338 read 0x02cd66 thru 0x030554 (0x37ef bytes) 000861 1769076546.938457 trunc from 0x038267 to 0x03a6fd (0x2497 bytes) 000862 1769076547.000029 fallocate 0x0385a3 thru 0x03ffff (0x7a5d bytes) 000864 1769076547.006350 mapread 0x02b746 thru 0x0379ef (0xc2aa bytes) 000865 1769076547.032245 read 0x0063ec thru 0x00a72f (0x4344 bytes) 000866 1769076547.042594 fallocate 0x034127 thru 0x03559d (0x1477 bytes) 000867 1769076547.047593 trunc from 0x040000 to 0x006cc9 (0x39336 bytes) 000868 1769076547.084062 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x6cca): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6cca): Mirror 16: 0xde569442 (stale) Mirror 17: 0xde569442 (stale) Mirror 19: 0xde569442 (stale) Mirror 21: 0xde569442 (stale) Mirror 23: 0xde569442 (stale) Mirror 24: 0xde569442 (stale) Mirror 25: 0xde569442 (stale) Mirror 26: 0xde569442 (stale) Mirror 27: 0x4ae2db10 (stale) Mirror 28: 0x4ae2db10 (stale) CRC-32 checksum value for chunk [0, 0x6cca): Mirror 1: 0x4ae2db10 Mirror 29: 0x4ae2db10 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6cca: skipped 000869 1769076547.863973 mapread 0x002da0 thru 0x006cc9 (0x3f2a bytes) 000870 1769076547.906809 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6cca): Mirror 16: 0xde569442 (stale) Mirror 17: 0xde569442 (stale) Mirror 19: 0xde569442 (stale) Mirror 21: 0xde569442 (stale) Mirror 23: 0xde569442 (stale) Mirror 24: 0xde569442 (stale) Mirror 25: 0xde569442 (stale) Mirror 26: 0xde569442 (stale) Mirror 27: 0x4ae2db10 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6cca): Mirror 16: 0xde569442 (stale) Mirror 17: 0xde569442 (stale) Mirror 19: 0xde569442 (stale) Mirror 21: 0xde569442 (stale) Mirror 23: 0xde569442 (stale) Mirror 24: 0xde569442 (stale) Mirror 25: 0xde569442 (stale) Mirror 26: 0xde569442 (stale) Mirror 27: 0x4ae2db10 (stale) CRC-32 checksum value for chunk [0, 0x6cca): Mirror 1: 0x4ae2db10 Mirror 29: 0x4ae2db10 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6cca: skipped 000871 1769076548.776249 read 0x004ac3 thru 0x006cc9 (0x2207 bytes) 000872 1769076548.799817 trunc from 0x006cca to 0x00b4df (0x4816 bytes) 000873 1769076548.890461 mapwrite 0x022ee7 thru 0x0295f6 (0x6710 bytes) 000874 1769076548.945361 trunc from 0x0241bf to 0x0295f6 (0x5438 bytes) 000875 1769076548.952629 trunc from 0x0240d6 to 0x0295f6 (0x5521 bytes) 000876 1769076548.958377 mapwrite 0x01fe56 thru 0x021ba8 (0x1d53 bytes) 000877 1769076548.971557 punch from 0x0168ee to 0x022127 (0xb83a bytes) 000878 1769076548.977129 trunc from 0x02414a to 0x025d95 (0x1c4c bytes) 000879 1769076548.981973 trunc from 0x0295f7 to 0x016f54 (0x126a2 bytes) 000880 1769076548.999297 fallocate 0x01a314 thru 0x01e81f (0x450c bytes) 000881 1769076549.007437 mapwrite 0x00d0d7 thru 0x014ca8 (0x7bd2 bytes) 000882 1769076549.032004 punch from 0x00047a to 0x0074d9 (0x7060 bytes) 000883 1769076549.037215 read 0x01d1e1 thru 0x01e81f (0x163f bytes) 000884 1769076549.051610 MIRROR_RESYNC 11 mirrors 000885 1769076551.446024 trunc from 0x01bf21 to 0x01e81f (0x28ff bytes) 000887 1769076551.509088 trunc from 0x011f62 to 0x01559f (0x363e bytes) 000889 1769076551.525438 mapread 0x00aa3f thru 0x00b95b (0xf1d bytes) 000890 1769076551.547373 trunc from 0x01e820 to 0x00e9ea (0xfe35 bytes) 000891 1769076551.601489 trunc from 0x002470 to 0x00a745 (0x82d6 bytes) 000892 1769076551.613446 mapread 0x0087b0 thru 0x00e9ea (0x623b bytes) 000893 1769076551.643939 fallocate 0x013cc3 thru 0x016326 (0x2664 bytes) 000894 1769076551.658354 MIRROR_RESYNC 11 mirrors 000895 1769076553.868943 read 0x004088 thru 0x00d8bb (0x9834 bytes) 000897 1769076553.882213 mapread 0x00d6e7 thru 0x00e9ea (0x1304 bytes) 000898 1769076553.888420 mapread 0x00dd8f thru 0x00e9ea (0xc5c bytes) 000899 1769076553.889151 read 0x006cc1 thru 0x00e9ea (0x7d2a bytes) 000900 1769076553.890127 write 0x03647e thru 0x03ffff (0x9b82 bytes) 000901 1769076553.982027 fallocate 0x018b40 thru 0x02264e (0x9b0f bytes) 000902 1769076554.005032 write 0x02a118 thru 0x031b60 (0x7a49 bytes) 000903 1769076554.038289 fallocate 0x003899 thru 0x01176f (0xded7 bytes) 000904 1769076554.046437 fallocate 0x036c61 thru 0x03c43b (0x57db bytes) 000905 1769076554.083373 read 0x00ac1b thru 0x00bce7 (0x10cd bytes) 000906 1769076554.095704 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xc5b7d92a (stale) Mirror 17: 0xc5b7d92a (stale) Mirror 21: 0xc5b7d92a (stale) Mirror 23: 0xc5b7d92a (stale) Mirror 24: 0xc5b7d92a (stale) Mirror 25: 0xc5b7d92a (stale) Mirror 26: 0xc5b7d92a (stale) Mirror 27: 0xc5b7d92a (stale) Mirror 29: 0xc5b7d92a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xc5b7d92a (stale) Mirror 17: 0xc5b7d92a (stale) Mirror 21: 0xc5b7d92a (stale) Mirror 23: 0xc5b7d92a (stale) Mirror 24: 0xc5b7d92a (stale) Mirror 25: 0xc5b7d92a (stale) Mirror 26: 0xc5b7d92a (stale) Mirror 27: 0xc5b7d92a (stale) Mirror 29: 0xc5b7d92a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000907 1769076554.905840 trunc from 0x040000 to 0x027bc1 (0x1843e bytes) 000908 1769076554.939275 trunc from 0x001b9a to 0x00cfe9 (0xb450 bytes) 000909 1769076554.946732 trunc from 0x0121b8 to 0x01e0e1 (0xbf2a bytes) 000910 1769076554.952577 write 0x00ecbc thru 0x01cec3 (0xe208 bytes) 000911 1769076554.966342 read 0x015234 thru 0x0210c7 (0xbe94 bytes) 000912 1769076554.977571 write 0x024fce thru 0x029ed9 (0x4f0c bytes) 000913 1769076554.998996 mapwrite 0x02fdce thru 0x037b8b (0x7dbe bytes) 000915 1769076555.151248 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37b8c): Mirror 16: 0xf40bbd6 (stale) Mirror 17: 0xf40bbd6 (stale) Mirror 21: 0xf40bbd6 (stale) Mirror 23: 0xf40bbd6 (stale) Mirror 25: 0xf40bbd6 (stale) Mirror 26: 0xf40bbd6 (stale) Mirror 27: 0xf40bbd6 (stale) Mirror 29: 0xf40bbd6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37b8c): Mirror 16: 0xf40bbd6 (stale) Mirror 17: 0xf40bbd6 (stale) Mirror 21: 0xf40bbd6 (stale) Mirror 23: 0xf40bbd6 (stale) Mirror 25: 0xf40bbd6 (stale) Mirror 26: 0xf40bbd6 (stale) Mirror 27: 0xf40bbd6 (stale) Mirror 29: 0xf40bbd6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37b8c: skipped 000916 1769076555.883649 fallocate 0x00a66d thru 0x017759 (0xd0ed bytes) 000917 1769076555.894974 write 0x01bf2e thru 0x02937d (0xd450 bytes) 000918 1769076555.954709 write 0x00e7a5 thru 0x0136bf (0x4f1b bytes) 000919 1769076555.996943 mapwrite 0x027782 thru 0x030e90 (0x970f bytes) 000920 1769076556.059286 write 0x032923 thru 0x03ea79 (0xc157 bytes) 000921 1769076556.063019 read 0x01225c thru 0x0209f1 (0xe796 bytes) 000922 1769076556.075570 read 0x01f8aa thru 0x02c879 (0xcfd0 bytes) 000923 1769076556.077954 write 0x0398f3 thru 0x03daac (0x41ba bytes) 000924 1769076556.079388 trunc from 0x039da8 to 0x03ea79 (0x4cd2 bytes) 000926 1769076556.101345 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3ea7a): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ea7a): Mirror 16: 0x711b4c66 (stale) Mirror 17: 0x711b4c66 (stale) Mirror 21: 0x711b4c66 (stale) Mirror 23: 0x711b4c66 (stale) Mirror 25: 0x711b4c66 (stale) Mirror 26: 0x711b4c66 (stale) Mirror 27: 0x711b4c66 (stale) Mirror 29: 0x711b4c66 (stale) CRC-32 checksum value for chunk [0, 0x3ea7a): Mirror 1: 0xa6dc88e4 Mirror 30: 0xa6dc88e4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ea7a: skipped 000928 1769076557.507000 trunc from 0x03ea7a to 0x03d1fe (0x187b bytes) 000929 1769076557.659541 trunc from 0x03d1ff to 0x009bc9 (0x33635 bytes) 000930 1769076557.687118 write 0x038e29 thru 0x03ffff (0x71d7 bytes) 000931 1769076557.688386 mapread 0x000530 thru 0x004add (0x45ae bytes) 000932 1769076557.724399 read 0x00f433 thru 0x01eec7 (0xfa95 bytes) 000933 1769076557.738948 punch from 0x01f8e5 to 0x02a44f (0xab6b bytes) 000934 1769076557.752340 punch from 0x0133b6 to 0x015bae (0x27f9 bytes) 000935 1769076557.761842 mapread 0x03a16a thru 0x03b344 (0x11db bytes) 000936 1769076557.763116 fallocate 0x031066 thru 0x038b30 (0x7acb bytes) 000937 1769076557.771418 punch from 0x01c499 to 0x02627c (0x9de4 bytes) 000938 1769076557.786817 mapwrite 0x033036 thru 0x035eb4 (0x2e7f bytes) 000939 1769076557.853975 fallocate 0x021783 thru 0x0252df (0x3b5d bytes) 000940 1769076557.859620 mapwrite 0x008b7f thru 0x0115aa (0x8a2c bytes) 000941 1769076557.894008 mapwrite 0x025b85 thru 0x03313b (0xd5b7 bytes) 000942 1769076557.937943 punch from 0x03476d to 0x036340 (0x1bd4 bytes) 000943 1769076557.951123 punch from 0x0087c5 to 0x010fa2 (0x87de bytes) 000944 1769076557.959724 punch from 0x0336aa to 0x039d5c (0x66b3 bytes) 000945 1769076557.979861 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x165:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xc5b7d92a (stale) Mirror 21: 0xc5b7d92a (stale) Mirror 23: 0xc5b7d92a (stale) Mirror 25: 0xc5b7d92a (stale) Mirror 26: 0xc5b7d92a (stale) Mirror 27: 0xc5b7d92a (stale) Mirror 29: 0xc5b7d92a (stale) Mirror 30: 0x8e5d54cb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xc5b7d92a (stale) Mirror 21: 0xc5b7d92a (stale) Mirror 23: 0xc5b7d92a (stale) Mirror 25: 0xc5b7d92a (stale) Mirror 26: 0xc5b7d92a (stale) Mirror 27: 0xc5b7d92a (stale) Mirror 29: 0xc5b7d92a (stale) Mirror 30: 0x8e5d54cb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000947 1769076558.783546 MIRROR_RESYNC 9 mirrors 000949 1769076562.665311 mapwrite 0x022758 thru 0x026c9f (0x4548 bytes) 000950 1769076562.763227 trunc from 0x024df8 to 0x029d49 (0x4f52 bytes) 000951 1769076562.773078 mapread 0x033412 thru 0x038f2c (0x5b1b bytes) 000952 1769076562.802317 MIRROR_RESYNC 9 mirrors 000953 1769076566.952982 trunc from 0x040000 to 0x03ba2f (0x45d0 bytes) 000954 1769076567.034108 trunc from 0x0242bb to 0x033c01 (0xf947 bytes) 000955 1769076567.062638 trunc from 0x03ba30 to 0x03380b (0x8224 bytes) 000956 1769076567.121136 mapread 0x0180d2 thru 0x02726c (0xf19b bytes) 000957 1769076567.225589 trunc from 0x03380c to 0x012ddc (0x20a2f bytes) 000958 1769076567.271534 write 0x00e056 thru 0x015fdb (0x7f86 bytes) 000959 1769076567.300968 punch from 0x004a13 to 0x013327 (0xe915 bytes) 000960 1769076567.327768 punch from 0x001cdf to 0x0083da (0x66fc bytes) 000961 1769076567.352673 mapread 0x009495 thru 0x015fdb (0xcb47 bytes) 000962 1769076567.379426 mapwrite 0x00c7a6 thru 0x01a022 (0xd87d bytes) 000963 1769076567.469125 read 0x00cb4a thru 0x012a76 (0x5f2d bytes) 000964 1769076567.470438 punch from 0x01030b to 0x015488 (0x517e bytes) 000965 1769076567.492140 fallocate 0x032cff thru 0x03ffff (0xd301 bytes) 000966 1769076567.504850 read 0x025e48 thru 0x035d42 (0xfefb bytes) 000968 1769076567.525717 trunc from 0x010200 to 0x01b5d0 (0xb3d1 bytes) 000969 1769076567.536845 read 0x034793 thru 0x036226 (0x1a94 bytes) 000970 1769076567.537996 write 0x03458d thru 0x039331 (0x4da5 bytes) 000971 1769076567.540080 fallocate 0x002779 thru 0x008a8e (0x6316 bytes) 000972 1769076567.556971 read 0x0125b2 thru 0x01ccc7 (0xa716 bytes) 000973 1769076567.595011 mapwrite 0x011d14 thru 0x018075 (0x6362 bytes) 000974 1769076567.625861 write 0x039891 thru 0x03ffff (0x676f bytes) 000976 1769076567.633257 mapwrite 0x033156 thru 0x03d0cb (0x9f76 bytes) 000979 1769076567.668347 read 0x0363a5 thru 0x03f77a (0x93d6 bytes) 000980 1769076567.669018 trunc from 0x02c3f2 to 0x02faf8 (0x3707 bytes) 000981 1769076567.682063 mapread 0x02458b thru 0x02ff8e (0xba04 bytes) 000982 1769076567.704534 mapwrite 0x01f26e thru 0x024740 (0x54d3 bytes) 000983 1769076567.742576 fallocate 0x02605e thru 0x02e498 (0x843b bytes) 000984 1769076567.753395 mapread 0x01309b thru 0x021af1 (0xea57 bytes) 000985 1769076567.754558 mapwrite 0x022a05 thru 0x02c85c (0x9e58 bytes) 000986 1769076567.796964 read 0x03439b thru 0x0352ad (0xf13 bytes) 000987 1769076567.797576 read 0x02f2b0 thru 0x03ec29 (0xf97a bytes) 000988 1769076567.798158 mapread 0x011d06 thru 0x012ba8 (0xea3 bytes) 000989 1769076567.798957 trunc from 0x040000 to 0x014e51 (0x2b1ae bytes) 000990 1769076567.831257 fallocate 0x03ed34 thru 0x03ffff (0x12cc bytes) 000991 1769076567.846019 mapwrite 0x00d579 thru 0x017ad8 (0xa560 bytes) 000992 1769076567.934149 write 0x03e7f9 thru 0x03ffff (0x1807 bytes) 000993 1769076567.949716 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xd292551c (stale) Mirror 21: 0xd292551c (stale) Mirror 23: 0xd292551c (stale) Mirror 25: 0xd292551c (stale) Mirror 26: 0xd292551c (stale) Mirror 27: 0xd292551c (stale) Mirror 29: 0xd292551c (stale) Mirror 30: 0xd292551c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb5d3fade Mirror 31: 0xb5d3fade lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000994 1769076569.584643 trunc from 0x040000 to 0x007964 (0x3869b bytes) 000995 1769076569.691169 punch from 0x007329 to 0x007964 (0x63c bytes) 000996 1769076569.711580 mapread 0x002242 thru 0x004671 (0x2430 bytes) 000997 1769076569.743195 mapread 0x0065a5 thru 0x007964 (0x13c0 bytes) 000998 1769076569.744101 trunc from 0x0069ac to 0x007964 (0xfb9 bytes) 000999 1769076569.756124 punch from 0x005d61 to 0x007964 (0x1c04 bytes) 001000 1769076569.774745 MIRROR_RESYNC 10 mirrors All operations completed A-OK! PASS 70a (161s) == sanity-flr test 71: check mirror extend parent fid ==== 05:09:38 (1769076578) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000404:0x1ad:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (14s) == sanity-flr test 72: check append on FLR file ========== 05:09:53 (1769076593) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x94:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x77:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.225921 s, 9.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x94:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x77:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132487 s, 7.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x94:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x78:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x77:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x94:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x78:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x77:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x96:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (13s) == sanity-flr test 200a: stress test ===================== 05:10:05 (1769076605) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4245536 .. Extending file size to 6740928 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7755136 .. Extending file size to 7909088 .. Extending file size to 7956384 .. done Extending file size to 8453344 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9172224 .. Extending file size to 8482080 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8694688 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 59758 59759 59762 59763 59766 failed 192.168.202.134@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre2 (opts:) 192.168.202.134@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (87s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 05:11:32 (1769076692) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4837088 .. Extending file size to 3041760 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8056672 .. failed Extending file size to 5676672 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8902720 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 5850848 .. Extending file size to 9118016 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8033888 .. Extending file size to 8129856 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8490592 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9313888 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8677440 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61541 61543 61544 61545 61548 done 192.168.202.134@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre2 (opts:) 192.168.202.134@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (84s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 05:12:56 (1769076776) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2280 1269024 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2264 1269040 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25000 3582020 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18944 3588076 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43944 7170096 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2280 1269024 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2264 1269040 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25000 3582020 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18944 3588076 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43944 7170096 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.106735 s, 19.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.236673 s, 8.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x93fdd77d (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.134@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre2 (opts:) PASS 200c (23s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 05:13:21 (1769076801) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.238106 s, 8.8 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 05:13:29 (1769076809) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x63:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 05:13:38 (1769076818) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113231 s, 18.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1f5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1f6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 05:13:48 (1769076828) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.115082 s, 18.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x91:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x91:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 05:13:58 (1769076838) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.27941 s, 7.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x92:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x92:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } PASS 204c (38s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 05:14:36 (1769076876) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg234-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132697 s, 7.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1f5:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1f7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1f5:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1f7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1f6:0x0] } PASS 204d (59s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 05:15:35 (1769076935) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg234-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1f7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1f8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.153765 s, 13.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1f7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1f8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1f9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1f7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1f8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1f9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (53s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 05:16:29 (1769076989) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg234-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg234-server: Pool lustre.test_204f created Adding targets to pool oleg234-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg234-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.147283 s, 14.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x93:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x93:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg234-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg234-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg234-server: Pool lustre.test_204f destroyed PASS 204f (83s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 05:17:52 (1769077072) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 05:18:01 (1769077081) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 05:18:11 (1769077091) oleg234-server: Pool lustre.test_206 created Adding targets to pool oleg234-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg234-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg234-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg234-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg234-server: Pool lustre.test_206 destroyed PASS 206 (34s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 05:18:44 (1769077124) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080325 s, 13.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674815 s, 15.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x708d9584 Mirror 3: 0x708d9584 PASS 207 (15s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 05:18:59 (1769077139) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 05:19:05 (1769077145) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (6s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 05:19:10 (1769077150) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0904957 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xb40000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1b70000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1b70000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1b70000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x950000 Read this (4 bytes): BBBB 192.168.202.134@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre2 (opts:) PASS 209a (43s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 05:19:53 (1769077193) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0158747 s, 516 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 05:20:01 (1769077201) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595252 s, 17.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x82 lmm_fid: [0x200000405:0x82:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x82 lmm_fid: [0x200000405:0x82:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbb:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611266 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-22 05:20:03.000000000 -0500 Modify: 2026-01-22 05:20:03.000000000 -0500 Change: 2026-01-22 05:20:05.000000000 -0500 Birth: 2026-01-22 05:20:03.000000000 -0500 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 05:20:10 (1769077210) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488946 s, 21.4 MB/s oleg234-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg234-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg234-server: O/280000401/d28/bc: File not found by ext2_lookup PASS 210b (23s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 05:20:33 (1769077233) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.255485 s, 160 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0263013 s, 156 kB/s size after second write -rw-r--r-- 1 root root 4096 Jan 22 05:20 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x87:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jan 22 05:20 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 05:20:42 (1769077242) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (10s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 05:20:52 (1769077252) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (12s) == sanity-flr test complete, duration 2111 sec =========== 05:21:04 (1769077264) === sanity-flr: start cleanup 05:21:06 (1769077266) === === sanity-flr: finish cleanup 05:21:15 (1769077275) ===