-----============= acceptance-small: sanity-flr ============----- Tue Apr 28 20:04:36 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg236-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 20:04:45 (1777421085) === oleg236-client.virtnet: executing check_config_client /mnt/lustre oleg236-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg236-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9f2e90b31000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f2e90b31000.idle_timeout=debug disable quota as required oleg236-server: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 20:04:59 (1777421099) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8175] == sanity-flr test 0a: lfs mirror create with -N option == 20:05:00 (1777421100) PASS 0a (6s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:05:06 (1777421106) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:05:07 (1777421107) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 20:05:09 (1777421109) PASS 0d (7s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:05:16 (1777421116) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:05:18 (1777421118) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 20:05:19 (1777421119) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9f2e90b31000.stats=clear osc.lustre-OST0001-osc-ffff9f2e90b31000.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:05:22 (1777421122) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:05:26 (1777421126) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:05:29 (1777421129) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:05:30 (1777421130) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 20:05:34 (1777421134) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:05:37 (1777421137) PASS 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:05:40 (1777421140) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:05:43 (1777421143) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:05:46 (1777421146) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:05:49 (1777421149) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 0.380913 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034996 s, 30.0 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:05:52 (1777421152) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.0769258 s, 81.8 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:05:55 (1777421155) Stopping /mnt/lustre-ost1 (opts:) on oleg236-server oleg236-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f2e90b31000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f2e90b31000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg236-server: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg236-server: oleg236-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg236-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg236-server oleg236-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f2e90b31000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f2e90b31000.ost_server_uuid in DISCONN state after 1 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg236-server: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg236-server: oleg236-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg236-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (27s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:06:22 (1777421182) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0614021 s, 51.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg236-server oleg236-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f2e90b31000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f2e90b31000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg236-server: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg236-server: oleg236-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg236-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (14s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:06:36 (1777421196) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg236-server oleg236-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f2e90b31000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f2e90b31000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg236-server: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg236-server: oleg236-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg236-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg236-server oleg236-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f2e90b31000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f2e90b31000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg236-server: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg236-server: oleg236-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg236-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (27s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:07:03 (1777421223) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.910779 s, 57.6 MB/s -rw-r--r-- 1 root root 52428800 Apr 28 20:07 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.253s user 0m0.000s sys 0m0.149s Stopping /mnt/lustre-ost1 (opts:) on oleg236-server oleg236-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f2e90b31000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f2e90b31000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.295s user 0m0.000s sys 0m0.157s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg236-server: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg236-server: oleg236-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg236-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg236-server oleg236-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f2e90b31000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f2e90b31000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.395s user 0m0.001s sys 0m0.157s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg236-server: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg236-server: oleg236-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg236-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg236-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f2e90b31000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f2e90b31000.ost_server_uuid in FULL state after 0 sec PASS 33b (28s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:07:31 (1777421251) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:07:32 (1777421252) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:07:34 (1777421254) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 20:07:35 (1777421255) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 20:07:38 (1777421258) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.176046 s, 71.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:07:42 (1777421262) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143322 s, 73.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013006 s, 80.6 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:07:44 (1777421264) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:07:48 (1777421268) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000953407 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0005679 s, 1.8 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000893612 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000736212 s, 1.4 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 20:07:51 (1777421271) create test files with size 15736607 .. f177dade6f2d0e0faa45d5a19cdd3618 - 7b1c7de09e739d9ff10bef9ca38351f1 - 59f2b43dba9b7f91a7b442d4984835ea - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg236-client.virtnet /mnt/lustre (opts:) Starting client: oleg236-client.virtnet: -o user_xattr,flock 192.168.202.136@tcp:/lustre /mnt/lustre PASS 37 (7s) == sanity-flr test 38: resync ============================ 20:07:58 (1777421278) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (6s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:08:04 (1777421284) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:08:07 (1777421287) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388375 s, 27.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130575 s, 80.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x41:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x40:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 20:08:10 (1777421290) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0330393 s, 63.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.14345 s, 29.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0724978 s, 28.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (6s) == sanity-flr test 42: lfs mirror verify ================= 20:08:16 (1777421296) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 20:08:17 (1777421297) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:08:19 (1777421299) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136083 s, 77.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 20:08:21 (1777421301) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0932953 s, 33.7 MB/s PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:08:24 (1777421304) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x49:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4b:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4b:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:08:28 (1777421308) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.111096 s, 94.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg236-client.virtnet /mnt/lustre (opts:) Starting client: oleg236-client.virtnet: -o user_xattr,flock 192.168.202.136@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (17s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:08:45 (1777421325) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9f2e85765000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.111427 s, 94.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg236-client.virtnet /mnt/lustre (opts:) Starting client: oleg236-client.virtnet: -o user_xattr,flock 192.168.202.136@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9f2e908c9000.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:08:50 (1777421330) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (15s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:09:05 (1777421345) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (14s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:09:19 (1777421359) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 20:09:20 (1777421360) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0907337 s, 34.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0678749 s, 46.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x72ec9024 (nosync,stale) Mirror 2: 0x72ec9024 (nosync,stale) ** verify mirror 2 doesn't change original checksum: dc511469ae6819c3465a815ba28992cf - mirror 1 checksum: 5e7ad3bd5e6906229e1900f9508a5142 - mirror 2 checksum: dc511469ae6819c3465a815ba28992cf - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Apr 28 20:09:20 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Apr 28 20:09:20 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: dc511469ae6819c3465a815ba28992cf - mirror 1 checksum: 5e7ad3bd5e6906229e1900f9508a5142 - mirror 2 checksum: 5e7ad3bd5e6906229e1900f9508a5142 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Apr 28 20:09:20 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Apr 28 20:09:20 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } PASS 48 (3s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 20:09:23 (1777421363) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140942 s, 74.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (3s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:09:26 (1777421366) Failing mds1 on oleg236-server Stopping /mnt/lustre-mds1 (opts:) on oleg236-server 20:09:33 (1777421373) shut down facet: mds1 facet_host: oleg236-server facet_failover_host: oleg236-server Failover mds1 to oleg236-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg236-server: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1 Started lustre-MDT0000 20:09:48 (1777421388) targets are mounted 20:09:48 (1777421388) facet_failover done oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164189 s, 63.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x56:0x0] } PASS 50A (31s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:09:57 (1777421397) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0154671 s, 1.3 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0109199 s, 1.9 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:10:01 (1777421401) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.014442 s, 1.4 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00694832 s, 2.9 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:10:04 (1777421404) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00322989 s, 5.1 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:10:07 (1777421407) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0218186 s, 939 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:10:10 (1777421410) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00280489 s, 1.5 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0177888 s, 230 kB/s PASS 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:10:13 (1777421413) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (9s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:10:22 (1777421422) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:10:42 (1777421442) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:11:00 (1777421460) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:11:18 (1777421478) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 20:11:21 (1777421481) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 55814 mirror split pid 55815 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4989: 55814 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4989: 55815 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 20:12:24 (1777421544) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 1545 skipping zero length zero range skipping zero length punch hole fallocating to largest ever: 0x334d7 000003 1777421545.817184 fallocate 0x02730e thru 0x0334d6 (0xc1c9 bytes) 000006 1777421545.824566 mapwrite 0x0099fb thru 0x0119c4 (0x7fca bytes) 000008 1777421545.851936 fallocate 0x0168d1 thru 0x021173 (0xa8a3 bytes) 000010 1777421545.856673 read 0x030994 thru 0x0334d6 (0x2b43 bytes) 000011 1777421545.861361 trunc from 0x0334d7 to 0x0312f7 (0x21df bytes) 000014 1777421545.877724 read 0x01d631 thru 0x02622f (0x8bff bytes) 000016 1777421545.884114 mapwrite 0x02cc4c thru 0x0399fe (0xcdb3 bytes) 000020 1777421545.929008 punch from 0x018459 to 0x020674 (0x821c bytes) 000024 1777421545.934573 write 0x00c06f thru 0x0184db (0xc46d bytes) 000026 1777421545.940668 read 0x00984b thru 0x009908 (0xbe bytes) zero_range to largest ever: 0x399ff 000027 1777421545.941271 trunc from 0x035c0b to 0x0399fe (0x3df4 bytes) 000028 1777421545.946393 punch from 0x0394e3 to 0x0399fe (0x51c bytes) 000029 1777421545.952439 trunc from 0x0399ff to 0x012196 (0x27868 bytes) 000030 1777421545.968474 punch from 0x003bd9 to 0x00c721 (0x8b49 bytes) 000031 1777421545.979874 trunc from 0x008507 to 0x00b3d4 (0x2ece bytes) 000032 1777421545.982571 read 0x0101fd thru 0x012196 (0x1f9a bytes) 000033 1777421545.983298 write 0x00c947 thru 0x00edf0 (0x24aa bytes) 000034 1777421545.993480 mapwrite 0x001e89 thru 0x004781 (0x28f9 bytes) 000035 1777421546.009042 mapread 0x005b1b thru 0x00d4d8 (0x79be bytes) 000036 1777421546.010344 mapread 0x001f17 thru 0x00dc2b (0xbd15 bytes) 000037 1777421546.011310 read 0x008d2d thru 0x012196 (0x946a bytes) 000038 1777421546.011893 mapread 0x002ed9 thru 0x00c4ca (0x95f2 bytes) 000039 1777421546.012850 mapread 0x004018 thru 0x0082a3 (0x428c bytes) 000040 1777421546.013830 punch from 0x011d55 to 0x012196 (0x442 bytes) 000041 1777421546.024369 write 0x0283cf thru 0x031f5f (0x9b91 bytes) 000042 1777421546.027972 MIRROR_EXTEND to 2 mirrors 000043 1777421546.429580 mapread 0x01e40b thru 0x02dfb6 (0xfbac bytes) 000044 1777421546.453529 mapread 0x00dee8 thru 0x01dac9 (0xfbe2 bytes) 000045 1777421546.462617 trunc from 0x00f623 to 0x01264d (0x302b bytes) 000046 1777421546.487906 trunc from 0x01904c to 0x0203e2 (0x7397 bytes) 000047 1777421546.490508 read 0x002504 thru 0x00a0ce (0x7bcb bytes) 000048 1777421546.496485 read 0x01d488 thru 0x02c55c (0xf0d5 bytes) 000049 1777421546.503093 fallocate 0x022bd6 thru 0x02c901 (0x9d2c bytes) 000050 1777421546.506439 mapwrite 0x007fa8 thru 0x0167bf (0xe818 bytes) 000051 1777421546.522756 fallocate 0x02a1f3 thru 0x02e950 (0x475e bytes) 000052 1777421546.526369 mapwrite 0x03f373 thru 0x03ffff (0xc8d bytes) 000053 1777421546.567983 mapread 0x01f09f thru 0x02c189 (0xd0eb bytes) 000054 1777421546.569117 trunc from 0x040000 to 0x001afe (0x3e501 bytes) 000055 1777421546.585580 mapwrite 0x02b33c thru 0x032b65 (0x782a bytes) 000056 1777421546.645745 fallocate 0x008f6b thru 0x00dc1e (0x4cb4 bytes) 000058 1777421546.653031 mapread 0x000c2b thru 0x008cd8 (0x80ae bytes) 000059 1777421546.682554 mapwrite 0x01f429 thru 0x02bece (0xcaa6 bytes) 000060 1777421546.734498 read 0x009076 thru 0x00e324 (0x52af bytes) 000062 1777421546.740538 read 0x008ce3 thru 0x0096d8 (0x9f6 bytes) 000063 1777421546.741079 punch from 0x02c94c to 0x02ec15 (0x22ca bytes) 000064 1777421546.744271 read 0x01b283 thru 0x02538d (0xa10b bytes) 000065 1777421546.744947 fallocate 0x00d924 thru 0x00fab9 (0x2196 bytes) 000066 1777421546.748814 punch from 0x002e94 to 0x00cab4 (0x9c21 bytes) 000068 1777421546.751949 fallocate 0x033f43 thru 0x036252 (0x2310 bytes) 000069 1777421546.754665 trunc from 0x00ea9f to 0x00f657 (0xbb9 bytes) 000070 1777421546.757388 mapwrite 0x00edea thru 0x015c45 (0x6e5c bytes) 000071 1777421546.776146 fallocate 0x0049eb thru 0x0127b5 (0xddcb bytes) 000072 1777421546.779277 mapread 0x00fd62 thru 0x01f008 (0xf2a7 bytes) 000073 1777421546.780519 write 0x022fb8 thru 0x028277 (0x52c0 bytes) 000074 1777421546.781558 trunc from 0x02ab40 to 0x03082f (0x5cf0 bytes) 000076 1777421546.788226 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36253): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36253): Mirror 2: 0x7b35730 (stale) CRC-32 checksum value for chunk [0, 0x36253): Mirror 1: 0xf53ea5df Mirror 3: 0xf53ea5df lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36253: skipped 000077 1777421547.422391 punch from 0x021969 to 0x0305a2 (0xec3a bytes) 000078 1777421547.454163 punch from 0x024bc1 to 0x02a1a0 (0x55e0 bytes) 000080 1777421547.457705 trunc from 0x02f171 to 0x036252 (0x70e2 bytes) 000081 1777421547.461865 write 0x002075 thru 0x00cefb (0xae87 bytes) 000082 1777421547.479472 punch from 0x00d231 to 0x0107e3 (0x35b3 bytes) 000083 1777421547.484862 trunc from 0x036253 to 0x01a514 (0x1bd3e bytes) 000084 1777421547.504682 punch from 0x0115ff to 0x01a514 (0x8f16 bytes) fallocating to largest ever: 0x3db99 000085 1777421547.509075 fallocate 0x034682 thru 0x03db98 (0x9517 bytes) 000087 1777421547.514018 mapread 0x006653 thru 0x009a12 (0x33c0 bytes) 000088 1777421547.515017 trunc from 0x00d0dd to 0x0183b0 (0xb2d4 bytes) 000089 1777421547.520667 write 0x03a6ad thru 0x03ba35 (0x1389 bytes) 000090 1777421547.522082 mapwrite 0x01a15b thru 0x023474 (0x931a bytes) 000091 1777421547.555054 punch from 0x034f90 to 0x03ba35 (0x6aa6 bytes) 000092 1777421547.564079 MIRROR_RESYNC 3 mirrors 000093 1777421548.135429 read 0x026387 thru 0x028398 (0x2012 bytes) 000094 1777421548.143541 MIRROR_RESYNC 3 mirrors 000095 1777421548.402190 mapread 0x03ba08 thru 0x03ba35 (0x2e bytes) 000096 1777421548.409250 mapwrite 0x011aa1 thru 0x012225 (0x785 bytes) 000097 1777421548.483373 read 0x02f179 thru 0x0307c4 (0x164c bytes) 000098 1777421548.488011 mapwrite 0x039249 thru 0x03ffff (0x6db7 bytes) 000099 1777421548.527744 write 0x011382 thru 0x018242 (0x6ec1 bytes) 000101 1777421548.529447 write 0x03c313 thru 0x03ffff (0x3ced bytes) 000102 1777421548.530487 read 0x01c2d8 thru 0x025cde (0x9a07 bytes) 000103 1777421548.541805 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xdbc86c65 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xdbc86c65 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000104 1777421548.720304 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xdbc86c65 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6580b257 Mirror 3: 0x6580b257 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000105 1777421549.196420 fallocate 0x02ebf3 thru 0x03b0e6 (0xc4f4 bytes) 000106 1777421549.249475 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xdbc86c65 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xdbc86c65 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000107 1777421549.440993 mapread 0x015ed3 thru 0x01e45e (0x858c bytes) 000108 1777421549.454240 trunc from 0x040000 to 0x03c05e (0x3fa1 bytes) 000109 1777421549.475827 trunc from 0x03c05f to 0x03bca7 (0x3b7 bytes) 000110 1777421549.484174 write 0x02a0be thru 0x02c034 (0x1f77 bytes) 000111 1777421549.491099 MIRROR_RESYNC 2 mirrors 000112 1777421549.918356 trunc from 0x03bca8 to 0x033cab (0x7ffc bytes) 000115 1777421549.950738 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x33cac): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33cac): Mirror 2: 0xe6361216 (stale) CRC-32 checksum value for chunk [0, 0x33cac): Mirror 1: 0xe6361216 Mirror 3: 0xe6361216 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33cac: skipped 000117 1777421550.547753 trunc from 0x033cac to 0x007dc7 (0x2bee4 bytes) 000118 1777421550.592577 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x7dc8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7dc8): Mirror 2: 0x41d44a11 (stale) Mirror 3: 0x41d44a11 (stale) CRC-32 checksum value for chunk [0, 0x7dc8): Mirror 1: 0x41d44a11 Mirror 4: 0x41d44a11 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7dc8: skipped 000120 1777421550.984433 trunc from 0x003796 to 0x0047b5 (0x1020 bytes) 000121 1777421551.008580 read 0x000d16 thru 0x007dc7 (0x70b2 bytes) 000122 1777421551.013663 read 0x002ce9 thru 0x007dc7 (0x50df bytes) 000124 1777421551.014971 read 0x00540b thru 0x007dc7 (0x29bd bytes) 000125 1777421551.015540 write 0x037545 thru 0x03ffff (0x8abb bytes) 000126 1777421551.020805 trunc from 0x040000 to 0x00a1a5 (0x35e5a bytes) 000127 1777421551.035499 punch from 0x0078aa to 0x009739 (0x1e90 bytes) 000128 1777421551.040107 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xa1a6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa1a6): Mirror 2: 0xd1bae341 (stale) Mirror 3: 0xd1bae341 (stale) Mirror 4: 0x200d962c (stale) CRC-32 checksum value for chunk [0, 0xa1a6): Mirror 1: 0x389e34fc Mirror 5: 0x389e34fc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa1a6: skipped truncating to largest ever: 0x3f4dd 000129 1777421551.576430 trunc from 0x00a1a6 to 0x03f4dc (0x35337 bytes) 000130 1777421551.641998 trunc from 0x03f4dd to 0x0060dc (0x39400 bytes) 000131 1777421551.652951 trunc from 0x00591d to 0x0060dc (0x7c0 bytes) 000132 1777421551.656363 trunc from 0x002d80 to 0x0060dc (0x335d bytes) 000133 1777421551.673394 trunc from 0x0060dd to 0x024fac (0x1eed0 bytes) 000134 1777421551.690578 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x24fad): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24fad): Mirror 2: 0x71920c99 (stale) Mirror 3: 0x71920c99 (stale) Mirror 4: 0xabb6a7f7 (stale) Mirror 5: 0xa4f0acec (stale) CRC-32 checksum value for chunk [0, 0x24fad): Mirror 1: 0xcdbc20cc Mirror 6: 0xcdbc20cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24fad: skipped 000136 1777421552.200508 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x24fad): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24fad): Mirror 2: 0x71920c99 (stale) Mirror 3: 0x71920c99 (stale) Mirror 4: 0xabb6a7f7 (stale) Mirror 5: 0xa4f0acec (stale) CRC-32 checksum value for chunk [0, 0x24fad): Mirror 1: 0xcdbc20cc Mirror 6: 0xcdbc20cc Mirror 7: 0xcdbc20cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24fad: skipped 000138 1777421552.708340 write 0x03e970 thru 0x03ffff (0x1690 bytes) 000139 1777421552.729384 MIRROR_RESYNC 7 mirrors 000140 1777421553.703128 punch from 0x032906 to 0x03ed06 (0xc401 bytes) 000141 1777421553.726538 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xdf05bc72 (stale) Mirror 3: 0xdf05bc72 (stale) Mirror 4: 0xdf05bc72 (stale) Mirror 5: 0xdf05bc72 (stale) Mirror 6: 0xdf05bc72 (stale) Mirror 7: 0xdf05bc72 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x69b6a1b3 Mirror 8: 0x69b6a1b3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000142 1777421554.302556 read 0x01f9a2 thru 0x021c1b (0x227a bytes) 000143 1777421554.308045 trunc from 0x040000 to 0x03927d (0x6d82 bytes) 000144 1777421554.339705 trunc from 0x00c8e1 to 0x00efc5 (0x26e5 bytes) 000145 1777421554.346416 trunc from 0x03927e to 0x025521 (0x13d5c bytes) 000146 1777421554.363983 read 0x01bf58 thru 0x021230 (0x52d9 bytes) 000147 1777421554.370615 mapwrite 0x02163c thru 0x02661e (0x4fe3 bytes) 000149 1777421554.391980 fallocate 0x011ed4 thru 0x013ac8 (0x1bf5 bytes) 000150 1777421554.394984 punch from 0x01906b to 0x02187a (0x8810 bytes) 000151 1777421554.398808 fallocate 0x0090e3 thru 0x016e2f (0xdd4d bytes) 000153 1777421554.406608 write 0x02e0a5 thru 0x03bd1c (0xdc78 bytes) 000154 1777421554.409135 write 0x014c16 thru 0x021dc1 (0xd1ac bytes) 000155 1777421554.416965 mapwrite 0x01841a thru 0x023f55 (0xbb3c bytes) 000156 1777421554.436313 trunc from 0x03bd1d to 0x020bfd (0x1b11f bytes) 000157 1777421554.446954 mapwrite 0x018408 thru 0x020b39 (0x8732 bytes) 000158 1777421554.463570 trunc from 0x020bfe to 0x01f53b (0x16c2 bytes) 000159 1777421554.475230 fallocate 0x007398 thru 0x015064 (0xdccd bytes) 000160 1777421554.480356 mapread 0x00a0f7 thru 0x00d6d0 (0x35da bytes) 000161 1777421554.481187 mapread 0x006b64 thru 0x0152fe (0xe79b bytes) 000163 1777421554.482589 write 0x026102 thru 0x029829 (0x3728 bytes) 000164 1777421554.483672 write 0x01d8aa thru 0x02206a (0x47c1 bytes) 000165 1777421554.488952 punch from 0x0245f4 to 0x026742 (0x214f bytes) 000166 1777421554.501905 mapread 0x0062a0 thru 0x009aef (0x3850 bytes) 000167 1777421554.503016 trunc from 0x0120f3 to 0x01a332 (0x8240 bytes) 000168 1777421554.507146 fallocate 0x030a8b thru 0x03a70b (0x9c81 bytes) 000170 1777421554.519015 read 0x016603 thru 0x0255d5 (0xefd3 bytes) truncating to largest ever: 0x3fd35 000171 1777421554.526306 trunc from 0x02982a to 0x03fd34 (0x1650b bytes) 000172 1777421554.533988 read 0x03dce1 thru 0x03fd34 (0x2054 bytes) 000173 1777421554.539614 trunc from 0x03097b to 0x03a88d (0x9f13 bytes) 000174 1777421554.543428 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3fd35): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd35): Mirror 2: 0x8128a8f3 (stale) Mirror 3: 0x8128a8f3 (stale) Mirror 4: 0x8128a8f3 (stale) Mirror 5: 0x8128a8f3 (stale) Mirror 6: 0x8128a8f3 (stale) Mirror 7: 0x8128a8f3 (stale) Mirror 8: 0x180beafd (stale) CRC-32 checksum value for chunk [0, 0x3fd35): Mirror 1: 0x2fc78634 Mirror 9: 0x2fc78634 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd35: skipped 000175 1777421555.082122 fallocate 0x032f93 thru 0x035097 (0x2105 bytes) 000176 1777421555.107298 fallocate 0x001fa5 thru 0x002dbb (0xe17 bytes) 000177 1777421555.110035 trunc from 0x016770 to 0x025f78 (0xf809 bytes) 000178 1777421555.112951 write 0x03d5df thru 0x03ffff (0x2a21 bytes) 000179 1777421555.120929 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xdf05bc72 (stale) Mirror 4: 0xdf05bc72 (stale) Mirror 5: 0xdf05bc72 (stale) Mirror 6: 0xdf05bc72 (stale) Mirror 7: 0xdf05bc72 (stale) Mirror 8: 0x69b6a1b3 (stale) Mirror 9: 0xa7158584 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xdf05bc72 (stale) Mirror 4: 0xdf05bc72 (stale) Mirror 5: 0xdf05bc72 (stale) Mirror 6: 0xdf05bc72 (stale) Mirror 7: 0xdf05bc72 (stale) Mirror 8: 0x69b6a1b3 (stale) Mirror 9: 0xa7158584 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000180 1777421555.424371 fallocate 0x0252c1 thru 0x025cae (0x9ee bytes) 000181 1777421555.428784 trunc from 0x0223ce to 0x02fdf6 (0xda29 bytes) 000183 1777421555.432973 punch from 0x021b02 to 0x02a8c1 (0x8dc0 bytes) 000184 1777421555.440412 MIRROR_RESYNC 8 mirrors 000185 1777421556.400972 punch from 0x023e5b to 0x02e064 (0xa20a bytes) 000186 1777421556.417985 write 0x00d17b thru 0x01743a (0xa2c0 bytes) 000188 1777421556.426401 trunc from 0x013375 to 0x021c0e (0xe89a bytes) 000189 1777421556.436331 mapwrite 0x012d01 thru 0x01ca63 (0x9d63 bytes) 000190 1777421556.455108 trunc from 0x040000 to 0x033581 (0xca7e bytes) 000191 1777421556.463936 fallocate 0x02a4d7 thru 0x02d081 (0x2bab bytes) 000192 1777421556.467497 mapread 0x00176b thru 0x00ed56 (0xd5ec bytes) 000193 1777421556.473147 trunc from 0x033582 to 0x02933b (0xa246 bytes) 000194 1777421556.480981 trunc from 0x0045af to 0x00b2f6 (0x6d48 bytes) 000195 1777421556.483737 read 0x000f9f thru 0x008355 (0x73b7 bytes) 000196 1777421556.487715 trunc from 0x02933c to 0x03aedb (0x11ba0 bytes) 000197 1777421556.498809 read 0x020db2 thru 0x026e94 (0x60e3 bytes) 000198 1777421556.508829 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3aedc): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aedc): Mirror 3: 0x560490d3 (stale) Mirror 4: 0x560490d3 (stale) Mirror 5: 0x560490d3 (stale) Mirror 6: 0x560490d3 (stale) Mirror 7: 0x560490d3 (stale) Mirror 8: 0x560490d3 (stale) Mirror 9: 0x560490d3 (stale) CRC-32 checksum value for chunk [0, 0x3aedc): Mirror 1: 0xf865b9ea Mirror 10: 0xf865b9ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aedc: skipped 000199 1777421556.995544 mapwrite 0x03db77 thru 0x03ffff (0x2489 bytes) 000200 1777421557.058959 punch from 0x02ddb9 to 0x035506 (0x774e bytes) 000201 1777421557.063445 trunc from 0x00d5b7 to 0x014821 (0x726b bytes) 000202 1777421557.067240 trunc from 0x040000 to 0x0389c2 (0x763d bytes) 000203 1777421557.076567 trunc from 0x010322 to 0x01793d (0x761c bytes) 000205 1777421557.079950 mapwrite 0x0299fc thru 0x031259 (0x785e bytes) 000206 1777421557.092297 trunc from 0x02a69f to 0x02ecfc (0x465e bytes) 000207 1777421557.097721 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x389c3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x389c3): Mirror 3: 0xce8b479a (stale) Mirror 4: 0xce8b479a (stale) Mirror 5: 0xce8b479a (stale) Mirror 6: 0xce8b479a (stale) Mirror 7: 0xce8b479a (stale) Mirror 8: 0xce8b479a (stale) Mirror 9: 0xce8b479a (stale) Mirror 10: 0xff6a90cb (stale) CRC-32 checksum value for chunk [0, 0x389c3): Mirror 1: 0x72671e29 Mirror 11: 0x72671e29 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x389c3: skipped 000208 1777421557.723124 write 0x036705 thru 0x03bb2c (0x5428 bytes) 000209 1777421557.764924 mapread 0x010816 thru 0x01e86c (0xe057 bytes) 000211 1777421557.783804 write 0x024d99 thru 0x029a09 (0x4c71 bytes) 000212 1777421557.790251 trunc from 0x016caa to 0x01c6d4 (0x5a2b bytes) 000213 1777421557.793085 fallocate 0x019d6b thru 0x0298f0 (0xfb86 bytes) 000214 1777421557.795805 fallocate 0x018d94 thru 0x025cc5 (0xcf32 bytes) 000215 1777421557.798682 punch from 0x027f54 to 0x037db3 (0xfe60 bytes) 000216 1777421557.818002 write 0x02b445 thru 0x033024 (0x7be0 bytes) 000217 1777421557.830558 MIRROR_RESYNC 10 mirrors 000218 1777421559.411007 read 0x019066 thru 0x020f5b (0x7ef6 bytes) 000219 1777421559.414974 mapwrite 0x01c4e6 thru 0x022351 (0x5e6c bytes) 000220 1777421559.464190 mapwrite 0x03f530 thru 0x03ffff (0xad0 bytes) 000221 1777421559.490545 fallocate 0x00bced thru 0x00f32d (0x3641 bytes) 000222 1777421559.494403 mapwrite 0x0331ca thru 0x03bae8 (0x891f bytes) 000223 1777421559.517423 trunc from 0x01cd54 to 0x02c56f (0xf81c bytes) 000224 1777421559.521514 fallocate 0x0205aa thru 0x02c00d (0xba64 bytes) 000225 1777421559.524603 trunc from 0x007ac0 to 0x014efd (0xd43e bytes) 000226 1777421559.527641 trunc from 0x013be2 to 0x01b685 (0x7aa4 bytes) 000227 1777421559.530673 mapwrite 0x00f2c2 thru 0x01ecd5 (0xfa14 bytes) 000229 1777421559.555259 trunc from 0x0295e2 to 0x029f35 (0x954 bytes) 000230 1777421559.558154 mapwrite 0x01b9c9 thru 0x025923 (0x9f5b bytes) 000231 1777421559.579584 trunc from 0x00441a to 0x005514 (0x10fb bytes) 000232 1777421559.583239 punch from 0x024d72 to 0x02be22 (0x70b1 bytes) 000233 1777421559.586397 punch from 0x005a27 to 0x00f0fa (0x96d4 bytes) 000234 1777421559.592154 punch from 0x006e37 to 0x01349d (0xc667 bytes) 000235 1777421559.595875 trunc from 0x040000 to 0x02e60b (0x119f4 bytes) 000237 1777421559.610635 mapread 0x003473 thru 0x00c00c (0x8b9a bytes) 000238 1777421559.616686 trunc from 0x01f729 to 0x02b473 (0xbd4b bytes) 000239 1777421559.622168 write 0x03bbde thru 0x03ffff (0x4422 bytes) 000240 1777421559.623527 trunc from 0x040000 to 0x0213a6 (0x1ec59 bytes) 000241 1777421559.636386 mapread 0x00dc26 thru 0x0119dc (0x3db7 bytes) 000243 1777421559.640972 trunc from 0x0213a7 to 0x005e48 (0x1b55e bytes) 000244 1777421559.649388 trunc from 0x004d14 to 0x005e48 (0x1135 bytes) 000245 1777421559.655906 read 0x003d8e thru 0x005e48 (0x20bb bytes) 000246 1777421559.661072 read 0x003373 thru 0x005e48 (0x2ad6 bytes) 000247 1777421559.661875 fallocate 0x018c99 thru 0x023b61 (0xaec9 bytes) 000248 1777421559.666748 read 0x001223 thru 0x0049ca (0x37a8 bytes) 000250 1777421559.667868 write 0x036a5b thru 0x03ffff (0x95a5 bytes) 000251 1777421559.669306 punch from 0x0260a3 to 0x0308c4 (0xa822 bytes) 000252 1777421559.672431 punch from 0x00c6fb to 0x01232c (0x5c32 bytes) 000253 1777421559.677561 read 0x03b288 thru 0x03ffff (0x4d78 bytes) 000254 1777421559.678291 fallocate 0x025c35 thru 0x026b10 (0xedc bytes) 000255 1777421559.681877 fallocate 0x011c62 thru 0x01a327 (0x86c6 bytes) 000256 1777421559.686576 fallocate 0x01975d thru 0x0283ed (0xec91 bytes) 000257 1777421559.689880 punch from 0x01af08 to 0x01f72d (0x4826 bytes) 000259 1777421559.693147 read 0x008345 thru 0x010ece (0x8b8a bytes) 000260 1777421559.699037 mapread 0x017a54 thru 0x0229f1 (0xaf9e bytes) 000261 1777421559.700477 punch from 0x01ff47 to 0x02eeda (0xef94 bytes) 000262 1777421559.704467 write 0x01f569 thru 0x02ac15 (0xb6ad bytes) 000263 1777421559.714111 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5bce9b5d (stale) Mirror 5: 0x5bce9b5d (stale) Mirror 6: 0x5bce9b5d (stale) Mirror 7: 0x5bce9b5d (stale) Mirror 8: 0x5bce9b5d (stale) Mirror 9: 0x5bce9b5d (stale) Mirror 10: 0x5bce9b5d (stale) Mirror 11: 0x5bce9b5d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5bce9b5d (stale) Mirror 5: 0x5bce9b5d (stale) Mirror 6: 0x5bce9b5d (stale) Mirror 7: 0x5bce9b5d (stale) Mirror 8: 0x5bce9b5d (stale) Mirror 9: 0x5bce9b5d (stale) Mirror 10: 0x5bce9b5d (stale) Mirror 11: 0x5bce9b5d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000264 1777421559.917730 write 0x032d34 thru 0x037a71 (0x4d3e bytes) 000266 1777421559.924138 mapwrite 0x025c9e thru 0x0340de (0xe441 bytes) 000268 1777421559.939291 trunc from 0x040000 to 0x00632e (0x39cd1 bytes) 000269 1777421559.944993 read 0x0046d2 thru 0x00632e (0x1c5d bytes) 000270 1777421559.947587 mapread 0x004097 thru 0x00632e (0x2298 bytes) 000271 1777421559.948548 mapread 0x002443 thru 0x0049fe (0x25bc bytes) 000272 1777421559.951421 read 0x001685 thru 0x00632e (0x4caa bytes) 000273 1777421559.953564 MIRROR_RESYNC 9 mirrors 000274 1777421560.474679 trunc from 0x00632f to 0x01b5e0 (0x152b2 bytes) 000275 1777421560.492625 trunc from 0x01b5e1 to 0x003a8e (0x17b52 bytes) 000276 1777421560.503400 mapread 0x002b39 thru 0x003a8e (0xf56 bytes) 000277 1777421560.514180 mapwrite 0x0284e2 thru 0x02f7dc (0x72fb bytes) 000278 1777421560.538307 trunc from 0x02f7dd to 0x025d50 (0x9a8c bytes) 000279 1777421560.551521 write 0x01f365 thru 0x025e24 (0x6ac0 bytes) 000280 1777421560.563075 read 0x007adf thru 0x00eeb9 (0x73db bytes) 000281 1777421560.568075 read 0x00e6b3 thru 0x01bdfa (0xd748 bytes) 000282 1777421560.572842 trunc from 0x006b28 to 0x0165da (0xfab3 bytes) 000283 1777421560.575819 trunc from 0x025e25 to 0x0065a9 (0x1f87b bytes) 000284 1777421560.584046 mapread 0x000946 thru 0x0065a9 (0x5c64 bytes) 000285 1777421560.587709 trunc from 0x002f5d to 0x0065a9 (0x364d bytes) 000286 1777421560.590768 trunc from 0x001dcb to 0x0065a9 (0x47df bytes) 000287 1777421560.593607 punch from 0x004ef3 to 0x0065a9 (0x16b7 bytes) 000289 1777421560.597726 trunc from 0x0012c2 to 0x0065a9 (0x52e8 bytes) 000290 1777421560.601446 read 0x00433a thru 0x0065a9 (0x2270 bytes) 000291 1777421560.605990 fallocate 0x00290d thru 0x005b1c (0x3210 bytes) 000292 1777421560.611684 MIRROR_RESYNC 9 mirrors 000293 1777421561.185482 fallocate 0x002ba2 thru 0x00690c (0x3d6b bytes) 000294 1777421561.208950 mapread 0x0040d2 thru 0x00690c (0x283b bytes) 000295 1777421561.229183 read 0x003161 thru 0x006507 (0x33a7 bytes) 000296 1777421561.229998 write 0x03fe51 thru 0x03ffff (0x1af bytes) 000297 1777421561.230910 trunc from 0x040000 to 0x00f59f (0x30a60 bytes) 000299 1777421561.251460 MIRROR_RESYNC 9 mirrors 000300 1777421561.939167 write 0x02c8af thru 0x03771c (0xae6e bytes) 000303 1777421561.974852 MIRROR_RESYNC 9 mirrors 000304 1777421562.782267 trunc from 0x02fc95 to 0x03771c (0x7a88 bytes) 000305 1777421562.803934 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3771d): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3771d): Mirror 4: 0x509c0b81 (stale) Mirror 5: 0x509c0b81 (stale) Mirror 6: 0x509c0b81 (stale) Mirror 7: 0x509c0b81 (stale) Mirror 8: 0x509c0b81 (stale) Mirror 9: 0x509c0b81 (stale) Mirror 10: 0x509c0b81 (stale) Mirror 11: 0x509c0b81 (stale) CRC-32 checksum value for chunk [0, 0x3771d): Mirror 1: 0xbcc123db Mirror 12: 0xbcc123db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3771d: skipped 000306 1777421563.372543 fallocate 0x00c975 thru 0x01a30b (0xd997 bytes) 000307 1777421563.392913 write 0x03ec1f thru 0x03ffff (0x13e1 bytes) 000308 1777421563.398120 read 0x01729b thru 0x0262cd (0xf033 bytes) 000311 1777421563.405772 mapread 0x020aea thru 0x021b77 (0x108e bytes) truncating to largest ever: 0x3ff14 000312 1777421563.406777 trunc from 0x040000 to 0x03ff13 (0xec bytes) 000313 1777421563.417153 mapwrite 0x017ecc thru 0x0278c4 (0xf9f9 bytes) 000314 1777421563.440105 punch from 0x012649 to 0x0141cc (0x1b84 bytes) fallocating to largest ever: 0x40000 000315 1777421563.445785 fallocate 0x039228 thru 0x03ffff (0x6dd8 bytes) 000316 1777421563.450076 mapread 0x03c062 thru 0x03cd28 (0xcc7 bytes) 000317 1777421563.453299 punch from 0x034eef to 0x03dea6 (0x8fb8 bytes) 000318 1777421563.455630 fallocate 0x0097c2 thru 0x00ceb7 (0x36f6 bytes) 000319 1777421563.464042 trunc from 0x03ff14 to 0x039d98 (0x617b bytes) 000320 1777421563.475662 mapread 0x02fa35 thru 0x039d98 (0xa364 bytes) 000321 1777421563.490365 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x39d99): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39d99): Mirror 4: 0xb790bb5 (stale) Mirror 5: 0xb790bb5 (stale) Mirror 6: 0xb790bb5 (stale) Mirror 7: 0xb790bb5 (stale) Mirror 8: 0xb790bb5 (stale) Mirror 9: 0xb790bb5 (stale) Mirror 10: 0xb790bb5 (stale) Mirror 11: 0xb790bb5 (stale) Mirror 12: 0x42c210fc (stale) CRC-32 checksum value for chunk [0, 0x39d99): Mirror 1: 0x7a8b45b4 Mirror 13: 0x7a8b45b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39d99: skipped 000322 1777421563.996853 read 0x01fb8b thru 0x0213c9 (0x183f bytes) 000323 1777421564.005882 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39d99): Mirror 4: 0xb790bb5 (stale) Mirror 5: 0xb790bb5 (stale) Mirror 6: 0xb790bb5 (stale) Mirror 7: 0xb790bb5 (stale) Mirror 8: 0xb790bb5 (stale) Mirror 10: 0xb790bb5 (stale) Mirror 11: 0xb790bb5 (stale) Mirror 12: 0x42c210fc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39d99): Mirror 4: 0xb790bb5 (stale) Mirror 5: 0xb790bb5 (stale) Mirror 6: 0xb790bb5 (stale) Mirror 7: 0xb790bb5 (stale) Mirror 8: 0xb790bb5 (stale) Mirror 10: 0xb790bb5 (stale) Mirror 11: 0xb790bb5 (stale) Mirror 12: 0x42c210fc (stale) CRC-32 checksum value for chunk [0, 0x39d99): Mirror 1: 0x7a8b45b4 Mirror 13: 0x7a8b45b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39d99: skipped 000324 1777421564.531170 write 0x0024b8 thru 0x00bbf8 (0x9741 bytes) 000325 1777421564.555390 fallocate 0x035360 thru 0x0364c1 (0x1162 bytes) 000326 1777421564.564091 fallocate 0x00f784 thru 0x01eb39 (0xf3b6 bytes) 000327 1777421564.567336 trunc from 0x039d99 to 0x02b228 (0xeb70 bytes) 000328 1777421564.583329 write 0x03ab43 thru 0x03ffff (0x54bd bytes) 000329 1777421564.584573 read 0x01425f thru 0x01c47d (0x821f bytes) 000330 1777421564.588435 trunc from 0x040000 to 0x01129b (0x2ed64 bytes) 000331 1777421564.601466 trunc from 0x01129c to 0x01b9fa (0xa75f bytes) 000332 1777421564.610906 write 0x026ca4 thru 0x03632f (0xf68c bytes) 000333 1777421564.613376 read 0x028049 thru 0x02f97c (0x7934 bytes) 000334 1777421564.614071 trunc from 0x030e73 to 0x03632f (0x54bd bytes) 000335 1777421564.621070 read 0x01569c thru 0x019369 (0x3cce bytes) 000336 1777421564.626420 mapwrite 0x038290 thru 0x03b695 (0x3406 bytes) 000337 1777421564.645621 read 0x01eb23 thru 0x0298c6 (0xada4 bytes) 000338 1777421564.646594 mapwrite 0x003253 thru 0x01007c (0xce2a bytes) 000339 1777421564.669528 mapread 0x01b99b thru 0x02b1ad (0xf813 bytes) 000340 1777421564.672874 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3b696): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b696): Mirror 4: 0xdb394f4d (stale) Mirror 5: 0xdb394f4d (stale) Mirror 6: 0xdb394f4d (stale) Mirror 7: 0xdb394f4d (stale) Mirror 8: 0xdb394f4d (stale) Mirror 10: 0xdb394f4d (stale) Mirror 11: 0xdb394f4d (stale) Mirror 12: 0x4db9cf5 (stale) Mirror 13: 0xe551498a (stale) CRC-32 checksum value for chunk [0, 0x3b696): Mirror 1: 0x2d8962a Mirror 14: 0x2d8962a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b696: skipped 000341 1777421565.130404 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3b696): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b696): Mirror 4: 0xdb394f4d (stale) Mirror 5: 0xdb394f4d (stale) Mirror 6: 0xdb394f4d (stale) Mirror 7: 0xdb394f4d (stale) Mirror 8: 0xdb394f4d (stale) Mirror 10: 0xdb394f4d (stale) Mirror 11: 0xdb394f4d (stale) Mirror 12: 0x4db9cf5 (stale) Mirror 13: 0xe551498a (stale) CRC-32 checksum value for chunk [0, 0x3b696): Mirror 1: 0x2d8962a Mirror 14: 0x2d8962a Mirror 15: 0x2d8962a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b696: skipped 000342 1777421565.557925 mapread 0x00a233 thru 0x00bfda (0x1da8 bytes) 000343 1777421565.562241 punch from 0x009e05 to 0x01313f (0x933b bytes) 000344 1777421565.579615 write 0x018582 thru 0x022165 (0x9be4 bytes) 000345 1777421565.586662 mapread 0x00ab59 thru 0x0148e2 (0x9d8a bytes) 000346 1777421565.590999 fallocate 0x01ca0a thru 0x01da88 (0x107f bytes) 000347 1777421565.593702 punch from 0x0008e7 to 0x00fab2 (0xf1cc bytes) 000348 1777421565.596579 trunc from 0x00f936 to 0x018ffe (0x96c9 bytes) 000349 1777421565.602991 trunc from 0x03b696 to 0x020b6a (0x1ab2b bytes) 000350 1777421565.620546 trunc from 0x01611f to 0x01a450 (0x4332 bytes) 000351 1777421565.623243 mapwrite 0x01f0d6 thru 0x01f680 (0x5ab bytes) 000352 1777421565.632953 fallocate 0x029208 thru 0x02980f (0x608 bytes) 000353 1777421565.637878 MIRROR_RESYNC 12 mirrors 000354 1777421566.220893 fallocate 0x01d1d5 thru 0x01e1f4 (0x1020 bytes) 000355 1777421566.236404 write 0x03a88b thru 0x03ffff (0x5775 bytes) 000356 1777421566.237682 trunc from 0x026754 to 0x02ba7f (0x532c bytes) 000357 1777421566.241493 read 0x02f166 thru 0x03b611 (0xc4ac bytes) 000358 1777421566.247447 MIRROR_RESYNC 12 mirrors 000359 1777421567.196963 trunc from 0x040000 to 0x01a6bb (0x25944 bytes) 000360 1777421567.233943 write 0x03f002 thru 0x03fc47 (0xc46 bytes) 000361 1777421567.235086 fallocate 0x01f7f5 thru 0x02e1d0 (0xe9dc bytes) 000362 1777421567.240710 write 0x01610b thru 0x01b103 (0x4ff9 bytes) 000363 1777421567.250754 MIRROR_RESYNC 12 mirrors 000364 1777421568.340022 trunc from 0x00fc63 to 0x0144e8 (0x4886 bytes) 000366 1777421568.366270 fallocate 0x0236a5 thru 0x0268a1 (0x31fd bytes) 000367 1777421568.370104 trunc from 0x03fc48 to 0x00498a (0x3b2bd bytes) 000368 1777421568.380941 punch from 0x003731 to 0x00498a (0x125a bytes) 000369 1777421568.383637 trunc from 0x00498b to 0x03701f (0x32695 bytes) 000370 1777421568.391866 fallocate 0x02401a thru 0x025e22 (0x1e09 bytes) 000371 1777421568.397176 read 0x01e904 thru 0x029086 (0xa783 bytes) 000372 1777421568.403124 punch from 0x02bb65 to 0x02c9d9 (0xe75 bytes) 000373 1777421568.405992 write 0x00903f thru 0x00e910 (0x58d2 bytes) 000374 1777421568.414425 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x37020): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37020): Mirror 4: 0x6375eeb0 (stale) Mirror 5: 0x6375eeb0 (stale) Mirror 6: 0x6375eeb0 (stale) Mirror 7: 0x6375eeb0 (stale) Mirror 8: 0x6375eeb0 (stale) Mirror 10: 0x6375eeb0 (stale) Mirror 11: 0x6375eeb0 (stale) Mirror 12: 0x6375eeb0 (stale) Mirror 13: 0x6375eeb0 (stale) Mirror 14: 0x6375eeb0 (stale) Mirror 15: 0x6375eeb0 (stale) CRC-32 checksum value for chunk [0, 0x37020): Mirror 1: 0xf4a39a43 Mirror 16: 0xf4a39a43 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37020: skipped 000375 1777421568.868818 write 0x03e07d thru 0x03ffff (0x1f83 bytes) 000376 1777421568.890516 fallocate 0x018567 thru 0x027dc3 (0xf85d bytes) 000377 1777421568.896889 mapwrite 0x012721 thru 0x01ad0c (0x85ec bytes) 000378 1777421568.922172 mapwrite 0x0376e4 thru 0x039a2a (0x2347 bytes) 000380 1777421568.933566 trunc from 0x040000 to 0x01180b (0x2e7f4 bytes) 000381 1777421568.950263 write 0x01e3f3 thru 0x02642a (0x8038 bytes) 000382 1777421568.951465 trunc from 0x02642b to 0x014c53 (0x117d7 bytes) 000383 1777421568.959403 read 0x00741a thru 0x00c9fe (0x55e5 bytes) 000384 1777421568.968688 mapread 0x0036df thru 0x0050ff (0x1a21 bytes) 000386 1777421568.980166 mapread 0x00a26f thru 0x00cd71 (0x2b03 bytes) 000387 1777421568.985154 punch from 0x011e1c to 0x014c53 (0x2e38 bytes) 000388 1777421568.990944 trunc from 0x014c54 to 0x00e494 (0x67bf bytes) 000389 1777421569.007362 trunc from 0x00e495 to 0x0057e9 (0x8cab bytes) 000391 1777421569.020369 write 0x0038a6 thru 0x004339 (0xa94 bytes) 000392 1777421569.021124 read 0x000d54 thru 0x00462c (0x38d9 bytes) 000393 1777421569.028076 fallocate 0x036f83 thru 0x03ffff (0x907d bytes) 000394 1777421569.032090 write 0x01ec9e thru 0x021825 (0x2b88 bytes) 000395 1777421569.033303 trunc from 0x021826 to 0x02c44d (0xac28 bytes) 000396 1777421569.052101 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c44e): Mirror 5: 0xa4c87261 (stale) Mirror 6: 0xa4c87261 (stale) Mirror 7: 0xa4c87261 (stale) Mirror 8: 0xa4c87261 (stale) Mirror 10: 0xa4c87261 (stale) Mirror 11: 0xa4c87261 (stale) Mirror 12: 0xa4c87261 (stale) Mirror 13: 0xa4c87261 (stale) Mirror 14: 0xa4c87261 (stale) Mirror 15: 0xa4c87261 (stale) Mirror 16: 0x44659720 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c44e): Mirror 5: 0xa4c87261 (stale) Mirror 6: 0xa4c87261 (stale) Mirror 7: 0xa4c87261 (stale) Mirror 8: 0xa4c87261 (stale) Mirror 10: 0xa4c87261 (stale) Mirror 11: 0xa4c87261 (stale) Mirror 12: 0xa4c87261 (stale) Mirror 13: 0xa4c87261 (stale) Mirror 14: 0xa4c87261 (stale) Mirror 15: 0xa4c87261 (stale) Mirror 16: 0x44659720 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c44e: skipped 000397 1777421569.357975 MIRROR_SPLIT mirror 6 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c44e): Mirror 5: 0xa4c87261 (stale) Mirror 6: 0xa4c87261 (stale) Mirror 7: 0xa4c87261 (stale) Mirror 8: 0xa4c87261 (stale) Mirror 10: 0xa4c87261 (stale) Mirror 12: 0xa4c87261 (stale) Mirror 13: 0xa4c87261 (stale) Mirror 14: 0xa4c87261 (stale) Mirror 15: 0xa4c87261 (stale) Mirror 16: 0x44659720 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c44e): Mirror 5: 0xa4c87261 (stale) Mirror 6: 0xa4c87261 (stale) Mirror 7: 0xa4c87261 (stale) Mirror 8: 0xa4c87261 (stale) Mirror 10: 0xa4c87261 (stale) Mirror 12: 0xa4c87261 (stale) Mirror 13: 0xa4c87261 (stale) Mirror 14: 0xa4c87261 (stale) Mirror 15: 0xa4c87261 (stale) Mirror 16: 0x44659720 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c44e: skipped 000398 1777421569.650441 write 0x037200 thru 0x03b03f (0x3e40 bytes) 000399 1777421569.660799 punch from 0x000f31 to 0x00f802 (0xe8d2 bytes) 000400 1777421569.668660 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3b040): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b040): Mirror 5: 0xd727eb7 (stale) Mirror 6: 0xd727eb7 (stale) Mirror 7: 0xd727eb7 (stale) Mirror 8: 0xd727eb7 (stale) Mirror 10: 0xd727eb7 (stale) Mirror 12: 0xd727eb7 (stale) Mirror 13: 0xd727eb7 (stale) Mirror 14: 0xd727eb7 (stale) Mirror 15: 0xd727eb7 (stale) Mirror 16: 0x5e17dfb9 (stale) CRC-32 checksum value for chunk [0, 0x3b040): Mirror 1: 0x48fe3127 Mirror 17: 0x48fe3127 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b040: skipped 000401 1777421570.183482 trunc from 0x029c31 to 0x039737 (0xfb07 bytes) 000402 1777421570.213575 MIRROR_SPLIT mirror 5 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b040): Mirror 5: 0xd727eb7 (stale) Mirror 6: 0xd727eb7 (stale) Mirror 7: 0xd727eb7 (stale) Mirror 8: 0xd727eb7 (stale) Mirror 12: 0xd727eb7 (stale) Mirror 13: 0xd727eb7 (stale) Mirror 14: 0xd727eb7 (stale) Mirror 15: 0xd727eb7 (stale) Mirror 16: 0x5e17dfb9 (stale) Mirror 17: 0x48fe3127 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b040): Mirror 5: 0xd727eb7 (stale) Mirror 6: 0xd727eb7 (stale) Mirror 7: 0xd727eb7 (stale) Mirror 8: 0xd727eb7 (stale) Mirror 12: 0xd727eb7 (stale) Mirror 13: 0xd727eb7 (stale) Mirror 14: 0xd727eb7 (stale) Mirror 15: 0xd727eb7 (stale) Mirror 16: 0x5e17dfb9 (stale) Mirror 17: 0x48fe3127 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b040: skipped 000403 1777421570.527887 fallocate 0x0020a6 thru 0x007e8f (0x5dea bytes) 000404 1777421570.532430 mapwrite 0x0300d3 thru 0x035a10 (0x593e bytes) 000405 1777421570.564324 read 0x026cd3 thru 0x033c1b (0xcf49 bytes) 000406 1777421570.572145 mapread 0x03a5ae thru 0x03b03f (0xa92 bytes) 000407 1777421570.577843 mapwrite 0x02b595 thru 0x02fc33 (0x469f bytes) 000408 1777421570.591127 fallocate 0x017825 thru 0x0203ee (0x8bca bytes) 000409 1777421570.594581 trunc from 0x0084c5 to 0x012602 (0xa13e bytes) 000410 1777421570.599886 mapread 0x02952f thru 0x032f6f (0x9a41 bytes) 000411 1777421570.605360 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b040): Mirror 5: 0xd727eb7 (stale) Mirror 6: 0xd727eb7 (stale) Mirror 7: 0xd727eb7 (stale) Mirror 8: 0xd727eb7 (stale) Mirror 12: 0xd727eb7 (stale) Mirror 14: 0xd727eb7 (stale) Mirror 15: 0xd727eb7 (stale) Mirror 16: 0x5e17dfb9 (stale) Mirror 17: 0x48fe3127 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b040): Mirror 5: 0xd727eb7 (stale) Mirror 6: 0xd727eb7 (stale) Mirror 7: 0xd727eb7 (stale) Mirror 8: 0xd727eb7 (stale) Mirror 12: 0xd727eb7 (stale) Mirror 14: 0xd727eb7 (stale) Mirror 15: 0xd727eb7 (stale) Mirror 16: 0x5e17dfb9 (stale) Mirror 17: 0x48fe3127 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b040: skipped 000412 1777421570.884645 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b040): Mirror 5: 0xd727eb7 (stale) Mirror 6: 0xd727eb7 (stale) Mirror 7: 0xd727eb7 (stale) Mirror 8: 0xd727eb7 (stale) Mirror 14: 0xd727eb7 (stale) Mirror 15: 0xd727eb7 (stale) Mirror 16: 0x5e17dfb9 (stale) Mirror 17: 0x48fe3127 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b040): Mirror 5: 0xd727eb7 (stale) Mirror 6: 0xd727eb7 (stale) Mirror 7: 0xd727eb7 (stale) Mirror 8: 0xd727eb7 (stale) Mirror 14: 0xd727eb7 (stale) Mirror 15: 0xd727eb7 (stale) Mirror 16: 0x5e17dfb9 (stale) Mirror 17: 0x48fe3127 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b040: skipped 000413 1777421571.125447 trunc from 0x03b040 to 0x0042bd (0x36d82 bytes) 000415 1777421571.146139 trunc from 0x003f7f to 0x0042bd (0x33f bytes) 000416 1777421571.154080 MIRROR_RESYNC 9 mirrors 000417 1777421571.687361 fallocate 0x00a5e4 thru 0x012f58 (0x8975 bytes) 000418 1777421571.712113 trunc from 0x0042be to 0x0214d1 (0x1d214 bytes) 000419 1777421571.731096 read 0x00d9f3 thru 0x01122b (0x3839 bytes) 000420 1777421571.736428 mapwrite 0x0290ef thru 0x02ae1c (0x1d2e bytes) 000421 1777421571.761408 trunc from 0x02ae1d to 0x011df2 (0x1902a bytes) 000422 1777421571.774013 write 0x024b25 thru 0x02b9eb (0x6ec7 bytes) 000423 1777421571.775482 mapread 0x000f2e thru 0x00d6a0 (0xc773 bytes) 000424 1777421571.785533 punch from 0x000e74 to 0x00a579 (0x9706 bytes) 000425 1777421571.788901 trunc from 0x028fd4 to 0x02b9eb (0x2a18 bytes) 000426 1777421571.801645 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b9ec): Mirror 5: 0x2f8a27d8 (stale) Mirror 6: 0x2f8a27d8 (stale) Mirror 7: 0x2f8a27d8 (stale) Mirror 8: 0x2f8a27d8 (stale) Mirror 14: 0x2f8a27d8 (stale) Mirror 16: 0x2f8a27d8 (stale) Mirror 17: 0x2f8a27d8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b9ec): Mirror 5: 0x2f8a27d8 (stale) Mirror 6: 0x2f8a27d8 (stale) Mirror 7: 0x2f8a27d8 (stale) Mirror 8: 0x2f8a27d8 (stale) Mirror 14: 0x2f8a27d8 (stale) Mirror 16: 0x2f8a27d8 (stale) Mirror 17: 0x2f8a27d8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b9ec: skipped 000427 1777421572.014392 trunc from 0x02b9ec to 0x030793 (0x4da8 bytes) 000428 1777421572.020254 punch from 0x01f7e4 to 0x0299d1 (0xa1ee bytes) 000429 1777421572.022451 trunc from 0x00eced to 0x010329 (0x163d bytes) 000430 1777421572.024254 mapread 0x014050 thru 0x015918 (0x18c9 bytes) 000431 1777421572.028427 read 0x013b3f thru 0x021f84 (0xe446 bytes) 000432 1777421572.034175 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30794): Mirror 6: 0x152bdc91 (stale) Mirror 7: 0x152bdc91 (stale) Mirror 8: 0x152bdc91 (stale) Mirror 14: 0x152bdc91 (stale) Mirror 16: 0x152bdc91 (stale) Mirror 17: 0x152bdc91 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30794): Mirror 6: 0x152bdc91 (stale) Mirror 7: 0x152bdc91 (stale) Mirror 8: 0x152bdc91 (stale) Mirror 14: 0x152bdc91 (stale) Mirror 16: 0x152bdc91 (stale) Mirror 17: 0x152bdc91 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30794: skipped 000433 1777421572.257852 read 0x013b3e thru 0x01a525 (0x69e8 bytes) 000434 1777421572.262126 mapwrite 0x032a84 thru 0x03bf98 (0x9515 bytes) 000435 1777421572.302114 trunc from 0x03bf99 to 0x02c0e8 (0xfeb0 bytes) 000436 1777421572.318392 punch from 0x020acb to 0x02b49b (0xa9d1 bytes) 000437 1777421572.322030 trunc from 0x02c0e9 to 0x02009d (0xc04b bytes) 000438 1777421572.334580 fallocate 0x02d30a thru 0x02f3d5 (0x20cc bytes) 000439 1777421572.337902 read 0x00f899 thru 0x01cc9e (0xd406 bytes) 000440 1777421572.343614 punch from 0x01261f to 0x013a6a (0x144c bytes) 000441 1777421572.346225 trunc from 0x02009e to 0x008e73 (0x1722a bytes) 000442 1777421572.364349 read 0x0030fb thru 0x008e73 (0x5d79 bytes) 000443 1777421572.364929 read 0x003098 thru 0x008e73 (0x5ddc bytes) 000445 1777421572.365831 write 0x03fa81 thru 0x03ffff (0x57f bytes) 000446 1777421572.366665 write 0x014efb thru 0x01b09a (0x61a0 bytes) 000447 1777421572.378715 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 14: 0xe20eea22 (stale) Mirror 16: 0xe20eea22 (stale) Mirror 17: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x352af8aa Mirror 18: 0x352af8aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000448 1777421572.811471 trunc from 0x0260ba to 0x03185e (0xb7a5 bytes) 000449 1777421572.834162 punch from 0x0357a1 to 0x036869 (0x10c9 bytes) 000450 1777421572.839917 MIRROR_RESYNC 8 mirrors 000451 1777421573.575092 read 0x01848e thru 0x025d36 (0xd8a9 bytes) 000452 1777421573.581368 trunc from 0x029d86 to 0x036069 (0xc2e4 bytes) 000453 1777421573.603962 trunc from 0x040000 to 0x007d74 (0x3828b bytes) 000454 1777421573.620870 mapread 0x00415f thru 0x00609f (0x1f41 bytes) 000455 1777421573.629402 trunc from 0x003ae8 to 0x007d74 (0x428d bytes) 000456 1777421573.633548 fallocate 0x0123c1 thru 0x01d9a4 (0xb5e4 bytes) 000457 1777421573.636742 trunc from 0x007d75 to 0x016396 (0xe622 bytes) 000458 1777421573.648392 trunc from 0x008def to 0x009436 (0x648 bytes) 000460 1777421573.652733 trunc from 0x016397 to 0x029836 (0x134a0 bytes) 000461 1777421573.662272 read 0x00736d thru 0x008059 (0xced bytes) 000462 1777421573.669913 trunc from 0x027149 to 0x029836 (0x26ee bytes) 000463 1777421573.673632 write 0x0044da thru 0x00623a (0x1d61 bytes) 000464 1777421573.674628 fallocate 0x000949 thru 0x003720 (0x2dd8 bytes) 000465 1777421573.678888 mapwrite 0x00dcc2 thru 0x011317 (0x3656 bytes) 000466 1777421573.690757 punch from 0x003967 to 0x00a25a (0x68f4 bytes) 000467 1777421573.701106 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29837): Mirror 7: 0x1e466fd1 (stale) Mirror 8: 0x1e466fd1 (stale) Mirror 14: 0x1e466fd1 (stale) Mirror 16: 0x1e466fd1 (stale) Mirror 17: 0x1e466fd1 (stale) Mirror 18: 0x1e466fd1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29837): Mirror 7: 0x1e466fd1 (stale) Mirror 8: 0x1e466fd1 (stale) Mirror 14: 0x1e466fd1 (stale) Mirror 16: 0x1e466fd1 (stale) Mirror 17: 0x1e466fd1 (stale) Mirror 18: 0x1e466fd1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29837: skipped 000468 1777421573.883851 fallocate 0x02d9ad thru 0x0333c4 (0x5a18 bytes) 000470 1777421573.887061 trunc from 0x00ddd7 to 0x011e54 (0x407e bytes) 000471 1777421573.891118 read 0x01cdd3 thru 0x01e486 (0x16b4 bytes) 000472 1777421573.895145 write 0x030779 thru 0x03abf3 (0xa47b bytes) 000473 1777421573.898105 mapwrite 0x01a138 thru 0x026ff6 (0xcebf bytes) 000474 1777421573.923943 mapwrite 0x034df8 thru 0x03a7fb (0x5a04 bytes) 000475 1777421573.951483 MIRROR_RESYNC 7 mirrors 000476 1777421575.239361 punch from 0x023703 to 0x02a8b5 (0x71b3 bytes) 000477 1777421575.252854 trunc from 0x03abf4 to 0x02fc4d (0xafa6 bytes) 000479 1777421575.264074 read 0x022f9a thru 0x029ba7 (0x6c0e bytes) 000480 1777421575.273400 write 0x01f576 thru 0x02e37a (0xee05 bytes) 000482 1777421575.282742 write 0x03a427 thru 0x03ffff (0x5bd9 bytes) 000483 1777421575.283547 mapwrite 0x017f09 thru 0x0217ea (0x98e2 bytes) 000484 1777421575.297390 mapread 0x007815 thru 0x00be9f (0x468b bytes) 000486 1777421575.302450 mapread 0x0371c3 thru 0x03ffff (0x8e3d bytes) 000487 1777421575.306768 trunc from 0x00517b to 0x011043 (0xbec9 bytes) 000489 1777421575.310427 mapwrite 0x01286f thru 0x018157 (0x58e9 bytes) 000490 1777421575.319594 mapread 0x019100 thru 0x01fa35 (0x6936 bytes) 000492 1777421575.320853 mapread 0x03b5dd thru 0x03ffff (0x4a23 bytes) 000493 1777421575.321504 mapread 0x00751b thru 0x00a8ab (0x3391 bytes) 000496 1777421575.327336 punch from 0x022ff2 to 0x029b80 (0x6b8f bytes) 000497 1777421575.336460 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe837b7f7 (stale) Mirror 8: 0xe837b7f7 (stale) Mirror 16: 0xe837b7f7 (stale) Mirror 17: 0xe837b7f7 (stale) Mirror 18: 0xe837b7f7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe837b7f7 (stale) Mirror 8: 0xe837b7f7 (stale) Mirror 16: 0xe837b7f7 (stale) Mirror 17: 0xe837b7f7 (stale) Mirror 18: 0xe837b7f7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000498 1777421575.540661 write 0x0264f8 thru 0x030851 (0xa35a bytes) 000500 1777421575.556082 mapread 0x028026 thru 0x029b51 (0x1b2c bytes) 000501 1777421575.556789 trunc from 0x040000 to 0x033676 (0xc989 bytes) 000502 1777421575.581358 punch from 0x010dd6 to 0x018641 (0x786c bytes) 000503 1777421575.589943 fallocate 0x02278b thru 0x02b926 (0x919c bytes) 000504 1777421575.599250 punch from 0x016fbb to 0x01e030 (0x7076 bytes) 000505 1777421575.607733 write 0x028e14 thru 0x036f4d (0xe13a bytes) 000506 1777421575.609547 write 0x006fd5 thru 0x00d530 (0x655c bytes) 000507 1777421575.618181 fallocate 0x015c34 thru 0x025c25 (0xfff2 bytes) 000508 1777421575.621791 write 0x019a89 thru 0x0207fa (0x6d72 bytes) 000509 1777421575.630948 punch from 0x0170fe to 0x022177 (0xb07a bytes) 000510 1777421575.641628 mapwrite 0x033529 thru 0x03c755 (0x922d bytes) 000511 1777421575.668463 trunc from 0x03a59f to 0x03c755 (0x21b7 bytes) 000513 1777421575.673668 mapread 0x026476 thru 0x02b6df (0x526a bytes) 000514 1777421575.674786 write 0x0272ea thru 0x02fbba (0x88d1 bytes) 000515 1777421575.676148 fallocate 0x026c01 thru 0x03673b (0xfb3b bytes) 000516 1777421575.679393 trunc from 0x00e33c to 0x01106b (0x2d30 bytes) 000517 1777421575.688123 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3c756): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c756): Mirror 7: 0x8ce5b41f (stale) Mirror 8: 0x8ce5b41f (stale) Mirror 16: 0x8ce5b41f (stale) Mirror 17: 0x8ce5b41f (stale) Mirror 18: 0x8ce5b41f (stale) CRC-32 checksum value for chunk [0, 0x3c756): Mirror 1: 0xdcffac96 Mirror 19: 0xdcffac96 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c756: skipped 000518 1777421576.178047 trunc from 0x02de4e to 0x0334ec (0x569f bytes) 000519 1777421576.207045 write 0x001f43 thru 0x00784e (0x590c bytes) 000520 1777421576.217637 write 0x03d690 thru 0x03ffff (0x2970 bytes) 000522 1777421576.219055 write 0x01c7ae thru 0x024c3f (0x8492 bytes) 000523 1777421576.227401 mapwrite 0x01b5bc thru 0x022ffb (0x7a40 bytes) 000524 1777421576.237792 trunc from 0x01cf81 to 0x0224f6 (0x5576 bytes) 000525 1777421576.240971 write 0x017360 thru 0x020aba (0x975b bytes) 000526 1777421576.247738 write 0x01e638 thru 0x02b025 (0xc9ee bytes) 000527 1777421576.258895 MIRROR_RESYNC 7 mirrors 000528 1777421577.341986 write 0x01da4a thru 0x02962a (0xbbe1 bytes) 000529 1777421577.361391 trunc from 0x040000 to 0x01e78e (0x21871 bytes) 000530 1777421577.373492 fallocate 0x0191da thru 0x02584a (0xc671 bytes) 000531 1777421577.378038 trunc from 0x00e49e to 0x0138b1 (0x5414 bytes) 000532 1777421577.381664 trunc from 0x01e78f to 0x02d76d (0xefdf bytes) 000533 1777421577.403783 fallocate 0x025506 thru 0x02ff81 (0xaa7c bytes) 000534 1777421577.407886 fallocate 0x02c474 thru 0x03340e (0x6f9b bytes) 000535 1777421577.411431 fallocate 0x028eb0 thru 0x02fdad (0x6efe bytes) 000536 1777421577.417352 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2fdae): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fdae): Mirror 7: 0xc4cf62c8 (stale) Mirror 8: 0xc4cf62c8 (stale) Mirror 16: 0xc4cf62c8 (stale) Mirror 17: 0xc4cf62c8 (stale) Mirror 18: 0xc4cf62c8 (stale) Mirror 19: 0xc4cf62c8 (stale) CRC-32 checksum value for chunk [0, 0x2fdae): Mirror 1: 0x718d093c Mirror 20: 0x718d093c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fdae: skipped 000537 1777421577.989093 read 0x012e3d thru 0x016927 (0x3aeb bytes) 000538 1777421577.995087 read 0x02fb84 thru 0x02fdad (0x22a bytes) 000539 1777421578.006236 fallocate 0x01f743 thru 0x02bac1 (0xc37f bytes) 000540 1777421578.037652 mapread 0x00f233 thru 0x0113b8 (0x2186 bytes) 000541 1777421578.057415 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2fdae): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fdae): Mirror 7: 0xc4cf62c8 (stale) Mirror 8: 0xc4cf62c8 (stale) Mirror 16: 0xc4cf62c8 (stale) Mirror 17: 0xc4cf62c8 (stale) Mirror 18: 0xc4cf62c8 (stale) Mirror 19: 0xc4cf62c8 (stale) Mirror 20: 0x718d093c (stale) CRC-32 checksum value for chunk [0, 0x2fdae): Mirror 1: 0x718d093c Mirror 21: 0x718d093c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fdae: skipped 000542 1777421578.626063 trunc from 0x02fdae to 0x005f72 (0x29e3b bytes) 000543 1777421578.650736 read 0x0051cb thru 0x005f72 (0xda8 bytes) 000544 1777421578.657931 mapread 0x002592 thru 0x0045ee (0x205d bytes) 000545 1777421578.670858 trunc from 0x005f73 to 0x0233bb (0x1d449 bytes) 000546 1777421578.690276 mapwrite 0x00b978 thru 0x01a272 (0xe8fb bytes) 000547 1777421578.725722 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x233bc): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x233bc): Mirror 7: 0x78a77bd (stale) Mirror 8: 0x78a77bd (stale) Mirror 16: 0x78a77bd (stale) Mirror 17: 0x78a77bd (stale) Mirror 18: 0x78a77bd (stale) Mirror 19: 0x78a77bd (stale) Mirror 20: 0xeb2ba726 (stale) Mirror 21: 0xeb2ba726 (stale) CRC-32 checksum value for chunk [0, 0x233bc): Mirror 1: 0xd7c0af67 Mirror 22: 0xd7c0af67 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x233bc: skipped 000548 1777421579.271942 read 0x018df8 thru 0x0233bb (0xa5c4 bytes) 000549 1777421579.282410 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x233bc): Mirror 7: 0x78a77bd (stale) Mirror 8: 0x78a77bd (stale) Mirror 17: 0x78a77bd (stale) Mirror 18: 0x78a77bd (stale) Mirror 19: 0x78a77bd (stale) Mirror 20: 0xeb2ba726 (stale) Mirror 21: 0xeb2ba726 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x233bc): Mirror 7: 0x78a77bd (stale) Mirror 8: 0x78a77bd (stale) Mirror 17: 0x78a77bd (stale) Mirror 18: 0x78a77bd (stale) Mirror 19: 0x78a77bd (stale) Mirror 20: 0xeb2ba726 (stale) Mirror 21: 0xeb2ba726 (stale) CRC-32 checksum value for chunk [0, 0x233bc): Mirror 1: 0xd7c0af67 Mirror 22: 0xd7c0af67 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x233bc: skipped 000550 1777421579.753054 write 0x02fcc6 thru 0x03ea36 (0xed71 bytes) 000551 1777421579.771435 mapwrite 0x0018b9 thru 0x00f3f8 (0xdb40 bytes) 000552 1777421579.789372 mapwrite 0x03c95b thru 0x03ffff (0x36a5 bytes) 000553 1777421579.819962 read 0x009912 thru 0x00d20c (0x38fb bytes) 000554 1777421579.820762 fallocate 0x03f3da thru 0x03ffff (0xc26 bytes) 000555 1777421579.824417 mapwrite 0x004669 thru 0x004708 (0xa0 bytes) 000556 1777421579.838129 trunc from 0x02279c to 0x023e43 (0x16a8 bytes) 000557 1777421579.842515 write 0x02f2c2 thru 0x03d578 (0xe2b7 bytes) 000558 1777421579.844094 mapread 0x0078fe thru 0x01447f (0xcb82 bytes) 000559 1777421579.854219 fallocate 0x01a402 thru 0x026fd2 (0xcbd1 bytes) 000560 1777421579.860433 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe2fd586c (stale) Mirror 8: 0xe2fd586c (stale) Mirror 17: 0xe2fd586c (stale) Mirror 18: 0xe2fd586c (stale) Mirror 19: 0xe2fd586c (stale) Mirror 20: 0xcf05f53b (stale) Mirror 21: 0xcf05f53b (stale) Mirror 22: 0xedb77572 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xebb2a6d1 Mirror 23: 0xebb2a6d1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000561 1777421580.480685 write 0x005354 thru 0x00f5c5 (0xa272 bytes) 000562 1777421580.506000 mapwrite 0x03aa7f thru 0x03ffff (0x5581 bytes) 000563 1777421580.520933 read 0x01a758 thru 0x022329 (0x7bd2 bytes) 000564 1777421580.527663 write 0x03fe62 thru 0x03ffff (0x19e bytes) 000565 1777421580.528768 write 0x03b98f thru 0x03ffff (0x4671 bytes) 000566 1777421580.531171 trunc from 0x040000 to 0x02a9dd (0x15622 bytes) 000567 1777421580.544598 read 0x020455 thru 0x02a677 (0xa223 bytes) 000568 1777421580.550285 punch from 0x016161 to 0x0234fe (0xd39e bytes) 000569 1777421580.553411 read 0x0166f5 thru 0x021839 (0xb145 bytes) 000570 1777421580.559750 mapwrite 0x035152 thru 0x03a73d (0x55ec bytes) 000571 1777421580.577737 trunc from 0x03a73e to 0x03696a (0x3dd3 bytes) 000572 1777421580.584052 trunc from 0x0000f4 to 0x002d33 (0x2c40 bytes) 000574 1777421580.587234 trunc from 0x02e9cb to 0x034b9f (0x61d5 bytes) 000575 1777421580.589904 fallocate 0x00a25a thru 0x011663 (0x740a bytes) 000576 1777421580.592121 fallocate 0x0260f4 thru 0x030cbf (0xabcc bytes) 000577 1777421580.594450 write 0x0360e0 thru 0x03ffff (0x9f20 bytes) 000579 1777421580.596348 mapwrite 0x009f60 thru 0x00f081 (0x5122 bytes) 000580 1777421580.605529 trunc from 0x040000 to 0x018973 (0x2768c bytes) 000581 1777421580.613592 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18974): Mirror 8: 0xd35e7de2 (stale) Mirror 17: 0xd35e7de2 (stale) Mirror 18: 0xd35e7de2 (stale) Mirror 19: 0xd35e7de2 (stale) Mirror 20: 0xd35e7de2 (stale) Mirror 21: 0xd35e7de2 (stale) Mirror 22: 0x32506b69 (stale) Mirror 23: 0xf0abb043 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18974): Mirror 8: 0xd35e7de2 (stale) Mirror 17: 0xd35e7de2 (stale) Mirror 18: 0xd35e7de2 (stale) Mirror 19: 0xd35e7de2 (stale) Mirror 20: 0xd35e7de2 (stale) Mirror 21: 0xd35e7de2 (stale) Mirror 22: 0x32506b69 (stale) Mirror 23: 0xf0abb043 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18974: skipped 000582 1777421580.896050 mapwrite 0x01d95f thru 0x0234f6 (0x5b98 bytes) 000583 1777421580.938100 mapwrite 0x0384d8 thru 0x03f52e (0x7057 bytes) 000587 1777421580.971085 trunc from 0x03db0f to 0x03f52e (0x1a20 bytes) 000588 1777421580.974589 mapread 0x026d68 thru 0x034067 (0xd300 bytes) 000589 1777421580.985244 read 0x005083 thru 0x00f3c1 (0xa33f bytes) 000590 1777421580.989755 write 0x03c0ca thru 0x03ffff (0x3f36 bytes) 000591 1777421580.991001 trunc from 0x040000 to 0x01ccc4 (0x2333b bytes) 000592 1777421581.001335 mapread 0x012ced thru 0x014c41 (0x1f55 bytes) 000593 1777421581.006876 trunc from 0x01ccc5 to 0x03fdae (0x230ea bytes) 000594 1777421581.016440 mapwrite 0x0000e5 thru 0x000d81 (0xc9d bytes) 000595 1777421581.027597 trunc from 0x03fdaf to 0x003323 (0x3ca8b bytes) 000596 1777421581.038859 read 0x001167 thru 0x001cfc (0xb96 bytes) 000597 1777421581.039467 mapread 0x002db4 thru 0x003323 (0x570 bytes) 000598 1777421581.040934 punch from 0x000b09 to 0x003323 (0x281b bytes) 000599 1777421581.043788 trunc from 0x003324 to 0x009643 (0x6320 bytes) 000600 1777421581.050548 fallocate 0x02570e thru 0x02aa93 (0x5386 bytes) 000602 1777421581.054665 punch from 0x0030c0 to 0x009643 (0x6584 bytes) 000603 1777421581.057357 punch from 0x004e72 to 0x005200 (0x38f bytes) 000604 1777421581.059688 punch from 0x00927e to 0x009643 (0x3c6 bytes) 000605 1777421581.061942 mapread 0x008308 thru 0x009643 (0x133c bytes) 000606 1777421581.067280 punch from 0x005ee6 to 0x009643 (0x375e bytes) 000607 1777421581.072274 punch from 0x0002b2 to 0x0037c3 (0x3512 bytes) 000608 1777421581.075245 punch from 0x008cfa to 0x0092ea (0x5f1 bytes) 000609 1777421581.078155 mapwrite 0x036bc2 thru 0x03ffff (0x943e bytes) 000611 1777421581.108024 read 0x0232b4 thru 0x03289b (0xf5e8 bytes) 000612 1777421581.108733 write 0x016506 thru 0x02398c (0xd487 bytes) 000613 1777421581.110426 punch from 0x00705e to 0x00baae (0x4a51 bytes) 000614 1777421581.114267 trunc from 0x040000 to 0x033776 (0xc889 bytes) 000615 1777421581.131289 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x33777): Mirror 8: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33777): Mirror 8: 0xa76965f4 (stale) Mirror 17: 0xa76965f4 (stale) Mirror 18: 0xa76965f4 (stale) Mirror 19: 0xa76965f4 (stale) Mirror 20: 0x26d079e1 (stale) Mirror 21: 0x26d079e1 (stale) Mirror 22: 0x8f634837 (stale) Mirror 23: 0xd61ca12 (stale) CRC-32 checksum value for chunk [0, 0x33777): Mirror 1: 0x775db184 Mirror 24: 0x775db184 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33777: skipped 000616 1777421581.661945 read 0x00fc74 thru 0x019a9c (0x9e29 bytes) 000617 1777421581.675561 mapwrite 0x030ea1 thru 0x03ada4 (0x9f04 bytes) 000618 1777421581.720561 write 0x039df3 thru 0x03ffff (0x620d bytes) 000619 1777421581.723471 MIRROR_RESYNC 10 mirrors 000620 1777421582.721850 punch from 0x035243 to 0x03ffff (0xadbd bytes) 000621 1777421582.739570 mapread 0x01e545 thru 0x02206d (0x3b29 bytes) 000623 1777421582.750768 trunc from 0x040000 to 0x00238d (0x3dc72 bytes) 000624 1777421582.766910 punch from 0x0007f6 to 0x00238d (0x1b98 bytes) 000626 1777421582.772781 read 0x000a47 thru 0x00238d (0x1947 bytes) 000627 1777421582.777940 trunc from 0x0002f2 to 0x00238d (0x209c bytes) 000628 1777421582.781192 punch from 0x001e7b to 0x00238d (0x513 bytes) 000630 1777421582.784864 fallocate 0x008b17 thru 0x00d363 (0x484d bytes) 000631 1777421582.788082 mapwrite 0x037563 thru 0x038f80 (0x1a1e bytes) 000632 1777421582.818417 mapread 0x02ff2e thru 0x036971 (0x6a44 bytes) 000634 1777421582.828381 fallocate 0x002155 thru 0x007f89 (0x5e35 bytes) 000635 1777421582.833872 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x38f81): Mirror 8: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f81): Mirror 8: 0xf3efe83c (stale) Mirror 17: 0xf3efe83c (stale) Mirror 18: 0xf3efe83c (stale) Mirror 19: 0xf3efe83c (stale) Mirror 20: 0xf3efe83c (stale) Mirror 21: 0xf3efe83c (stale) Mirror 22: 0xf3efe83c (stale) Mirror 23: 0xf3efe83c (stale) Mirror 24: 0xf3efe83c (stale) CRC-32 checksum value for chunk [0, 0x38f81): Mirror 1: 0x70f3746b Mirror 25: 0x70f3746b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f81: skipped 000636 1777421583.272047 mapwrite 0x024b08 thru 0x031065 (0xc55e bytes) 000637 1777421583.345524 mapread 0x031a64 thru 0x038f80 (0x751d bytes) 000638 1777421583.349938 mapread 0x031c4f thru 0x037e84 (0x6236 bytes) 000639 1777421583.350869 trunc from 0x0247a2 to 0x025048 (0x8a7 bytes) 000640 1777421583.353662 trunc from 0x038f81 to 0x00adb5 (0x2e1cb bytes) 000641 1777421583.370582 trunc from 0x000663 to 0x0088eb (0x8289 bytes) 000642 1777421583.374638 trunc from 0x00adb6 to 0x0099be (0x13f7 bytes) 000643 1777421583.382190 read 0x005679 thru 0x0099be (0x4346 bytes) 000644 1777421583.386279 fallocate 0x027e35 thru 0x02e03f (0x620b bytes) 000645 1777421583.389537 mapread 0x022b22 thru 0x024098 (0x1577 bytes) 000646 1777421583.394277 write 0x032962 thru 0x03ffff (0xd69e bytes) 000647 1777421583.395687 mapwrite 0x00a703 thru 0x00fbe2 (0x54e0 bytes) 000648 1777421583.413838 write 0x016356 thru 0x01f659 (0x9304 bytes) 000649 1777421583.418969 write 0x0296a6 thru 0x032264 (0x8bbf bytes) 000650 1777421583.420566 punch from 0x01b03e to 0x02a8e0 (0xf8a3 bytes) 000651 1777421583.431860 trunc from 0x040000 to 0x03d7ec (0x2813 bytes) 000652 1777421583.440982 mapread 0x0327af thru 0x03d7ec (0xb03e bytes) 000653 1777421583.442288 write 0x02f570 thru 0x03adfd (0xb88e bytes) 000654 1777421583.444229 fallocate 0x01d285 thru 0x02a73c (0xd4b8 bytes) 000656 1777421583.451736 write 0x0152b8 thru 0x01775b (0x24a4 bytes) 000657 1777421583.452818 write 0x01478a thru 0x0190d3 (0x494a bytes) 000658 1777421583.454017 mapwrite 0x01760c thru 0x01fcfb (0x86f0 bytes) 000659 1777421583.474365 trunc from 0x03d7ed to 0x03e8c3 (0x10d7 bytes) 000660 1777421583.484430 fallocate 0x037a9a thru 0x038b38 (0x109f bytes) 000661 1777421583.490976 trunc from 0x03e8c4 to 0x02e170 (0x10753 bytes) 000662 1777421583.501781 write 0x008af3 thru 0x00a597 (0x1aa5 bytes) 000663 1777421583.503071 trunc from 0x01b7dc to 0x028421 (0xcc46 bytes) 000664 1777421583.507488 read 0x0134b0 thru 0x013f72 (0xac3 bytes) 000665 1777421583.508279 read 0x025611 thru 0x028a40 (0x3430 bytes) 000666 1777421583.516979 punch from 0x001f01 to 0x00bd49 (0x9e49 bytes) 000667 1777421583.524348 read 0x029f02 thru 0x029fed (0xec bytes) 000668 1777421583.525119 punch from 0x00b0ba to 0x00b238 (0x17f bytes) 000669 1777421583.527810 write 0x02e899 thru 0x031f84 (0x36ec bytes) 000670 1777421583.528957 fallocate 0x01adba thru 0x026316 (0xb55d bytes) 000671 1777421583.533301 trunc from 0x022dff to 0x02f80c (0xca0e bytes) 000672 1777421583.540933 write 0x033f31 thru 0x03d92b (0x99fb bytes) 000673 1777421583.542610 read 0x00341f thru 0x00e54e (0xb130 bytes) 000674 1777421583.551054 fallocate 0x00c116 thru 0x01a4fe (0xe3e9 bytes) 000675 1777421583.553543 punch from 0x01822b to 0x02546d (0xd243 bytes) 000676 1777421583.556253 read 0x02b4f0 thru 0x02d3c0 (0x1ed1 bytes) 000677 1777421583.556765 trunc from 0x03d92c to 0x019b15 (0x23e16 bytes) 000678 1777421583.566026 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19b16): Mirror 8: 0x618d6ffc (stale) Mirror 17: 0x618d6ffc (stale) Mirror 19: 0x618d6ffc (stale) Mirror 20: 0x618d6ffc (stale) Mirror 21: 0x618d6ffc (stale) Mirror 22: 0x618d6ffc (stale) Mirror 23: 0x618d6ffc (stale) Mirror 24: 0x618d6ffc (stale) Mirror 25: 0xb11c0600 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19b16): Mirror 8: 0x618d6ffc (stale) Mirror 17: 0x618d6ffc (stale) Mirror 19: 0x618d6ffc (stale) Mirror 20: 0x618d6ffc (stale) Mirror 21: 0x618d6ffc (stale) Mirror 22: 0x618d6ffc (stale) Mirror 23: 0x618d6ffc (stale) Mirror 24: 0x618d6ffc (stale) Mirror 25: 0xb11c0600 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19b16: skipped 000679 1777421583.814446 trunc from 0x014ed7 to 0x019b15 (0x4c3f bytes) 000680 1777421583.818067 mapread 0x00e505 thru 0x019b15 (0xb611 bytes) 000681 1777421583.834244 read 0x0160c5 thru 0x0164b9 (0x3f5 bytes) 000682 1777421583.834974 fallocate 0x027527 thru 0x035545 (0xe01f bytes) 000683 1777421583.838273 punch from 0x00fcfa to 0x0157b2 (0x5ab9 bytes) 000684 1777421583.841035 write 0x01bd84 thru 0x028516 (0xc793 bytes) 000685 1777421583.846966 punch from 0x00a192 to 0x010135 (0x5fa4 bytes) 000686 1777421583.850230 mapread 0x016d15 thru 0x019fa1 (0x328d bytes) 000687 1777421583.851364 mapread 0x0266f9 thru 0x028516 (0x1e1e bytes) 000688 1777421583.852181 write 0x017baa thru 0x01999a (0x1df1 bytes) 000689 1777421583.853664 read 0x01881d thru 0x024866 (0xc04a bytes) 000690 1777421583.858279 trunc from 0x028517 to 0x010e3f (0x176d7 bytes) 000691 1777421583.871514 trunc from 0x010e40 to 0x025e97 (0x15058 bytes) 000692 1777421583.886848 fallocate 0x017fb6 thru 0x0240dc (0xc127 bytes) 000693 1777421583.892956 fallocate 0x03da63 thru 0x03ffff (0x259d bytes) 000694 1777421583.898783 fallocate 0x027b9d thru 0x02cdd0 (0x5234 bytes) 000695 1777421583.902017 read 0x018dbd thru 0x026b21 (0xdd65 bytes) 000696 1777421583.910335 punch from 0x00bdde to 0x013ad5 (0x7cf8 bytes) 000697 1777421583.915163 mapwrite 0x00acfb thru 0x010c1e (0x5f24 bytes) 000698 1777421583.938815 fallocate 0x02c048 thru 0x03b483 (0xf43c bytes) 000699 1777421583.941893 trunc from 0x040000 to 0x03c646 (0x39b9 bytes) 000700 1777421583.948451 trunc from 0x004dce to 0x00adad (0x5fe0 bytes) 000701 1777421583.951154 write 0x03f62f thru 0x03ffff (0x9d1 bytes) 000702 1777421583.952355 fallocate 0x009ccc thru 0x018242 (0xe577 bytes) 000703 1777421583.955629 read 0x0077a7 thru 0x015be9 (0xe443 bytes) 000704 1777421583.962888 MIRROR_RESYNC 10 mirrors 000705 1777421584.669631 trunc from 0x040000 to 0x03195e (0xe6a1 bytes) 000706 1777421584.682644 read 0x0165fb thru 0x01e16c (0x7b72 bytes) 000708 1777421584.686576 fallocate 0x02757c thru 0x031ef7 (0xa97c bytes) 000710 1777421584.689121 fallocate 0x01f29b thru 0x02cbaf (0xd915 bytes) 000711 1777421584.696605 MIRROR_RESYNC 10 mirrors 000712 1777421585.383419 read 0x016af3 thru 0x01ce5a (0x6368 bytes) 000714 1777421585.387510 fallocate 0x01bc53 thru 0x01c8ab (0xc59 bytes) 000716 1777421585.404683 mapwrite 0x01dfdb thru 0x01ee9a (0xec0 bytes) 000717 1777421585.422993 mapwrite 0x02740c thru 0x036e7b (0xfa70 bytes) 000718 1777421585.470810 trunc from 0x036e7c to 0x031d0b (0x5170 bytes) 000719 1777421585.480880 trunc from 0x016d2c to 0x017f22 (0x11f7 bytes) 000720 1777421585.488073 write 0x02a87c thru 0x035d43 (0xb4c8 bytes) 000722 1777421585.490563 write 0x00365b thru 0x005a60 (0x2406 bytes) 000723 1777421585.499650 read 0x0228be thru 0x025843 (0x2f86 bytes) 000724 1777421585.500449 mapwrite 0x030084 thru 0x0310a9 (0x1026 bytes) 000725 1777421585.509653 mapread 0x0064f2 thru 0x00f7aa (0x92b9 bytes) 000726 1777421585.510630 trunc from 0x024dcd to 0x02ea3d (0x9c71 bytes) 000727 1777421585.514075 write 0x027069 thru 0x0358af (0xe847 bytes) 000728 1777421585.518761 mapread 0x01c526 thru 0x02a17d (0xdc58 bytes) 000729 1777421585.519684 read 0x01d8d2 thru 0x02b522 (0xdc51 bytes) 000730 1777421585.520253 mapread 0x027e6b thru 0x030ded (0x8f83 bytes) 000731 1777421585.521302 mapread 0x02b915 thru 0x035cf8 (0xa3e4 bytes) 000732 1777421585.522408 read 0x000f9a thru 0x0099d2 (0x8a39 bytes) 000734 1777421585.523328 mapread 0x0327e2 thru 0x035d43 (0x3562 bytes) 000735 1777421585.524243 trunc from 0x01ba9f to 0x0222be (0x6820 bytes) 000736 1777421585.527648 trunc from 0x035d44 to 0x019c69 (0x1c0da bytes) 000737 1777421585.534116 fallocate 0x01a250 thru 0x01fc60 (0x5a11 bytes) 000738 1777421585.536837 mapwrite 0x00c1d3 thru 0x00ed4c (0x2b7a bytes) 000739 1777421585.542357 mapwrite 0x027819 thru 0x02ad31 (0x3519 bytes) 000740 1777421585.560085 write 0x002f11 thru 0x004a12 (0x1b02 bytes) 000741 1777421585.563705 MIRROR_RESYNC 10 mirrors 000742 1777421586.345502 read 0x01bd3f thru 0x0282e4 (0xc5a6 bytes) 000743 1777421586.349898 write 0x01d29a thru 0x023fc0 (0x6d27 bytes) 000744 1777421586.368469 mapwrite 0x0349ab thru 0x03ffff (0xb655 bytes) 000745 1777421586.392337 read 0x032eb5 thru 0x03322f (0x37b bytes) 000746 1777421586.393121 mapwrite 0x013a46 thru 0x01f0b3 (0xb66e bytes) 000747 1777421586.407326 trunc from 0x03aa3e to 0x03ffff (0x55c2 bytes) 000748 1777421586.410465 trunc from 0x040000 to 0x01969c (0x26963 bytes) 000749 1777421586.418074 trunc from 0x0137e1 to 0x01969c (0x5ebc bytes) 000750 1777421586.420581 trunc from 0x018439 to 0x01969c (0x1264 bytes) 000751 1777421586.422634 trunc from 0x010487 to 0x011ac3 (0x163d bytes) 000753 1777421586.425104 mapread 0x00c564 thru 0x00f98e (0x342b bytes) 000755 1777421586.426710 trunc from 0x01969d to 0x0076c0 (0x11fdc bytes) 000756 1777421586.435297 MIRROR_RESYNC 10 mirrors 000758 1777421586.987967 trunc from 0x0076c1 to 0x033b47 (0x2c487 bytes) 000759 1777421587.014530 write 0x00bd7a thru 0x0120eb (0x6372 bytes) 000761 1777421587.021085 write 0x01615c thru 0x0193db (0x3280 bytes) 000762 1777421587.028684 trunc from 0x033b48 to 0x016e84 (0x1ccc3 bytes) 000763 1777421587.042138 read 0x00b3fd thru 0x012b90 (0x7794 bytes) 000764 1777421587.042968 mapwrite 0x03eb70 thru 0x03ffff (0x1490 bytes) 000765 1777421587.073051 fallocate 0x03d59b thru 0x03ffff (0x2a65 bytes) 000766 1777421587.080907 fallocate 0x007616 thru 0x011fe3 (0xa9ce bytes) 000767 1777421587.084110 punch from 0x022287 to 0x031dc8 (0xfb42 bytes) 000768 1777421587.087109 fallocate 0x01fad0 thru 0x027521 (0x7a52 bytes) 000769 1777421587.094414 write 0x03c3fb thru 0x03ffff (0x3c05 bytes) 000770 1777421587.095286 read 0x00c1ca thru 0x015afd (0x9934 bytes) 000771 1777421587.095993 read 0x030c3d thru 0x037427 (0x67eb bytes) 000772 1777421587.105863 read 0x02b493 thru 0x03176e (0x62dc bytes) 000773 1777421587.106424 read 0x039960 thru 0x03ffff (0x66a0 bytes) 000774 1777421587.106952 punch from 0x0301d8 to 0x0316a5 (0x14ce bytes) 000775 1777421587.110122 punch from 0x0072c1 to 0x00ea01 (0x7741 bytes) 000777 1777421587.116689 mapwrite 0x02e3f3 thru 0x034291 (0x5e9f bytes) 000779 1777421587.146531 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xd69c9629 (stale) Mirror 17: 0xd69c9629 (stale) Mirror 19: 0xd69c9629 (stale) Mirror 20: 0xd69c9629 (stale) Mirror 21: 0xd69c9629 (stale) Mirror 22: 0xd69c9629 (stale) Mirror 24: 0xd69c9629 (stale) Mirror 25: 0xd69c9629 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xd69c9629 (stale) Mirror 17: 0xd69c9629 (stale) Mirror 19: 0xd69c9629 (stale) Mirror 20: 0xd69c9629 (stale) Mirror 21: 0xd69c9629 (stale) Mirror 22: 0xd69c9629 (stale) Mirror 24: 0xd69c9629 (stale) Mirror 25: 0xd69c9629 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000781 1777421587.427918 mapread 0x00d3e1 thru 0x01c9b7 (0xf5d7 bytes) 000782 1777421587.444758 mapread 0x03ced4 thru 0x03ffff (0x312c bytes) 000783 1777421587.450235 mapread 0x03ac40 thru 0x03ffff (0x53c0 bytes) 000784 1777421587.451128 trunc from 0x040000 to 0x01e1f8 (0x21e07 bytes) 000785 1777421587.463899 mapread 0x011aaa thru 0x01429b (0x27f2 bytes) 000786 1777421587.465406 write 0x01a172 thru 0x0209ee (0x687d bytes) 000789 1777421587.467952 write 0x032a50 thru 0x038511 (0x5ac2 bytes) 000790 1777421587.469261 write 0x029e05 thru 0x02ac3e (0xe3a bytes) 000791 1777421587.474113 mapwrite 0x005058 thru 0x0106fe (0xb6a7 bytes) 000792 1777421587.493365 trunc from 0x02585a to 0x028df2 (0x3599 bytes) 000793 1777421587.498246 fallocate 0x01a9d7 thru 0x0243bf (0x99e9 bytes) 000794 1777421587.503246 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38512): Mirror 8: 0xc6c26025 (stale) Mirror 17: 0xc6c26025 (stale) Mirror 20: 0xc6c26025 (stale) Mirror 21: 0xc6c26025 (stale) Mirror 22: 0xc6c26025 (stale) Mirror 24: 0xc6c26025 (stale) Mirror 25: 0xc6c26025 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38512): Mirror 8: 0xc6c26025 (stale) Mirror 17: 0xc6c26025 (stale) Mirror 20: 0xc6c26025 (stale) Mirror 21: 0xc6c26025 (stale) Mirror 22: 0xc6c26025 (stale) Mirror 24: 0xc6c26025 (stale) Mirror 25: 0xc6c26025 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38512: skipped 000795 1777421587.711310 mapread 0x011a1c thru 0x0154f2 (0x3ad7 bytes) 000796 1777421587.720299 punch from 0x0149b4 to 0x01d998 (0x8fe5 bytes) 000797 1777421587.723406 punch from 0x00486a to 0x00e02f (0x97c6 bytes) 000798 1777421587.728259 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x38512): Mirror 8: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38512): Mirror 8: 0xc6c26025 (stale) Mirror 17: 0xc6c26025 (stale) Mirror 20: 0xc6c26025 (stale) Mirror 21: 0xc6c26025 (stale) Mirror 22: 0xc6c26025 (stale) Mirror 24: 0xc6c26025 (stale) Mirror 25: 0xc6c26025 (stale) CRC-32 checksum value for chunk [0, 0x38512): Mirror 1: 0x7cc34ef Mirror 26: 0x7cc34ef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38512: skipped 000799 1777421588.199923 write 0x00ba2e thru 0x0108df (0x4eb2 bytes) 000801 1777421588.221878 fallocate 0x00a72d thru 0x0138d3 (0x91a7 bytes) 000802 1777421588.225453 punch from 0x01fef6 to 0x028c48 (0x8d53 bytes) 000803 1777421588.233428 MIRROR_RESYNC 9 mirrors 000804 1777421589.476917 trunc from 0x0248e1 to 0x02a7f1 (0x5f11 bytes) 000805 1777421589.498010 mapread 0x02939a thru 0x02fb9b (0x6802 bytes) 000806 1777421589.512545 write 0x02aefb thru 0x02f458 (0x455e bytes) 000807 1777421589.513508 mapwrite 0x03fe60 thru 0x03ffff (0x1a0 bytes) 000809 1777421589.543072 mapread 0x000354 thru 0x00f9c4 (0xf671 bytes) 000812 1777421589.549901 write 0x014d1f thru 0x016675 (0x1957 bytes) 000813 1777421589.563140 punch from 0x012976 to 0x01631e (0x39a9 bytes) 000814 1777421589.574760 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x130e8ada (stale) Mirror 17: 0x130e8ada (stale) Mirror 20: 0x130e8ada (stale) Mirror 22: 0x130e8ada (stale) Mirror 24: 0x130e8ada (stale) Mirror 25: 0x130e8ada (stale) Mirror 26: 0x130e8ada (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x130e8ada (stale) Mirror 17: 0x130e8ada (stale) Mirror 20: 0x130e8ada (stale) Mirror 22: 0x130e8ada (stale) Mirror 24: 0x130e8ada (stale) Mirror 25: 0x130e8ada (stale) Mirror 26: 0x130e8ada (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000815 1777421589.838641 read 0x022001 thru 0x02cf58 (0xaf58 bytes) 000816 1777421589.845074 trunc from 0x040000 to 0x0022d0 (0x3dd2f bytes) 000817 1777421589.860498 MIRROR_RESYNC 8 mirrors 000819 1777421590.361300 punch from 0x0014cf to 0x0022d0 (0xe02 bytes) 000820 1777421590.375892 fallocate 0x0098e9 thru 0x011d75 (0x848d bytes) 000821 1777421590.380622 MIRROR_RESYNC 8 mirrors 000822 1777421590.886432 punch from 0x00180d to 0x0022d0 (0xac4 bytes) 000823 1777421590.901084 mapread 0x001b82 thru 0x0022d0 (0x74f bytes) 000824 1777421590.906564 write 0x01c4f7 thru 0x02aa6e (0xe578 bytes) 000825 1777421590.908354 fallocate 0x02a740 thru 0x038073 (0xd934 bytes) 000826 1777421590.911174 write 0x01dcd6 thru 0x025c21 (0x7f4c bytes) 000827 1777421590.912545 write 0x03378a thru 0x03ffff (0xc876 bytes) 000828 1777421590.917246 fallocate 0x024dc3 thru 0x027ace (0x2d0c bytes) 000829 1777421590.920469 trunc from 0x02f641 to 0x0311eb (0x1bab bytes) 000830 1777421590.924570 read 0x00bcab thru 0x013490 (0x77e6 bytes) 000831 1777421590.929400 mapread 0x02843b thru 0x034608 (0xc1ce bytes) 000832 1777421590.933928 punch from 0x0130e7 to 0x01a7b8 (0x76d2 bytes) 000833 1777421590.940029 trunc from 0x040000 to 0x003536 (0x3cac9 bytes) 000834 1777421590.950665 mapwrite 0x008869 thru 0x00d746 (0x4ede bytes) 000835 1777421590.973453 mapread 0x001507 thru 0x0091a6 (0x7ca0 bytes) 000836 1777421590.974795 punch from 0x0006d0 to 0x00a298 (0x9bc9 bytes) 000837 1777421590.977717 fallocate 0x00ad74 thru 0x00d361 (0x25ee bytes) 000838 1777421590.980058 mapread 0x00bb2e thru 0x00d746 (0x1c19 bytes) 000839 1777421590.982979 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd747): Mirror 17: 0x6f02fbe (stale) Mirror 20: 0x6f02fbe (stale) Mirror 22: 0x6f02fbe (stale) Mirror 24: 0x6f02fbe (stale) Mirror 25: 0x6f02fbe (stale) Mirror 26: 0x6f02fbe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd747): Mirror 17: 0x6f02fbe (stale) Mirror 20: 0x6f02fbe (stale) Mirror 22: 0x6f02fbe (stale) Mirror 24: 0x6f02fbe (stale) Mirror 25: 0x6f02fbe (stale) Mirror 26: 0x6f02fbe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd747: skipped 000840 1777421591.170455 mapwrite 0x003f2e thru 0x006940 (0x2a13 bytes) 000841 1777421591.189545 write 0x033092 thru 0x03a81e (0x778d bytes) 000842 1777421591.193466 trunc from 0x03a81f to 0x01b27c (0x1f5a2 bytes) 000843 1777421591.200090 mapread 0x00d4de thru 0x017043 (0x9b66 bytes) 000844 1777421591.209586 trunc from 0x001847 to 0x010b18 (0xf2d2 bytes) 000845 1777421591.215034 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1b27d): Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b27d): Mirror 17: 0xb8b72e00 (stale) Mirror 20: 0xb8b72e00 (stale) Mirror 22: 0xb8b72e00 (stale) Mirror 24: 0xb8b72e00 (stale) Mirror 25: 0xb8b72e00 (stale) Mirror 26: 0xb8b72e00 (stale) CRC-32 checksum value for chunk [0, 0x1b27d): Mirror 1: 0xb8b72e00 Mirror 27: 0xb8b72e00 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b27d: skipped 000846 1777421591.574762 write 0x001c8a thru 0x008090 (0x6407 bytes) 000847 1777421591.591812 fallocate 0x02aaea thru 0x02d213 (0x272a bytes) 000848 1777421591.594379 trunc from 0x0181ab to 0x027069 (0xeebf bytes) 000849 1777421591.596890 mapwrite 0x02b68e thru 0x02cdf6 (0x1769 bytes) 000850 1777421591.609865 mapread 0x0006dc thru 0x001bb3 (0x14d8 bytes) 000851 1777421591.615373 mapwrite 0x0309a3 thru 0x03ee40 (0xe49e bytes) 000852 1777421591.653776 trunc from 0x00069c to 0x00df4f (0xd8b4 bytes) 000853 1777421591.656887 read 0x023ca7 thru 0x02f0a4 (0xb3fe bytes) 000854 1777421591.660775 write 0x01d85d thru 0x027ad0 (0xa274 bytes) 000855 1777421591.665769 mapread 0x002752 thru 0x007f6f (0x581e bytes) 000856 1777421591.670152 punch from 0x02b104 to 0x0359f9 (0xa8f6 bytes) 000857 1777421591.674951 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3ee41): Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ee41): Mirror 17: 0x895cb7f9 (stale) Mirror 20: 0x895cb7f9 (stale) Mirror 22: 0x895cb7f9 (stale) Mirror 24: 0x895cb7f9 (stale) Mirror 25: 0x895cb7f9 (stale) Mirror 26: 0x895cb7f9 (stale) Mirror 27: 0x895cb7f9 (stale) CRC-32 checksum value for chunk [0, 0x3ee41): Mirror 1: 0x374a48da Mirror 28: 0x374a48da lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ee41: skipped 000858 1777421592.110593 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3ee41): Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ee41): Mirror 17: 0x895cb7f9 (stale) Mirror 20: 0x895cb7f9 (stale) Mirror 22: 0x895cb7f9 (stale) Mirror 24: 0x895cb7f9 (stale) Mirror 25: 0x895cb7f9 (stale) Mirror 26: 0x895cb7f9 (stale) Mirror 27: 0x895cb7f9 (stale) CRC-32 checksum value for chunk [0, 0x3ee41): Mirror 1: 0x374a48da Mirror 28: 0x374a48da Mirror 29: 0x374a48da lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ee41: skipped 000859 1777421592.575977 write 0x011f9e thru 0x014271 (0x22d4 bytes) 000860 1777421592.604340 read 0x02a8a3 thru 0x02c1a5 (0x1903 bytes) 000861 1777421592.610613 punch from 0x035c1e to 0x03ee40 (0x9223 bytes) 000862 1777421592.614187 write 0x02a35d thru 0x02e020 (0x3cc4 bytes) 000863 1777421592.619103 mapwrite 0x031396 thru 0x03d19e (0xbe09 bytes) 000864 1777421592.648577 read 0x014a7e thru 0x017d3c (0x32bf bytes) 000865 1777421592.654852 trunc from 0x01700b to 0x01be49 (0x4e3f bytes) 000866 1777421592.659838 punch from 0x036c3f to 0x03ee40 (0x8202 bytes) 000867 1777421592.665577 fallocate 0x01e01d thru 0x029778 (0xb75c bytes) 000868 1777421592.669821 trunc from 0x01248d to 0x01d45c (0xafd0 bytes) 000869 1777421592.682420 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ee41): Mirror 17: 0x895cb7f9 (stale) Mirror 22: 0x895cb7f9 (stale) Mirror 24: 0x895cb7f9 (stale) Mirror 25: 0x895cb7f9 (stale) Mirror 26: 0x895cb7f9 (stale) Mirror 27: 0x895cb7f9 (stale) Mirror 28: 0x374a48da (stale) Mirror 29: 0x374a48da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ee41): Mirror 17: 0x895cb7f9 (stale) Mirror 22: 0x895cb7f9 (stale) Mirror 24: 0x895cb7f9 (stale) Mirror 25: 0x895cb7f9 (stale) Mirror 26: 0x895cb7f9 (stale) Mirror 27: 0x895cb7f9 (stale) Mirror 28: 0x374a48da (stale) Mirror 29: 0x374a48da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ee41: skipped 000870 1777421592.963975 trunc from 0x03ee41 to 0x0185b8 (0x26888 bytes) 000871 1777421592.984877 mapwrite 0x00ebf6 thru 0x00f0bc (0x4c7 bytes) 000872 1777421593.004265 fallocate 0x028e64 thru 0x02e21c (0x53b9 bytes) 000873 1777421593.007496 fallocate 0x01142d thru 0x01560a (0x41de bytes) 000874 1777421593.011157 mapread 0x006ce4 thru 0x011a23 (0xad40 bytes) 000875 1777421593.021779 fallocate 0x009002 thru 0x00aee8 (0x1ee7 bytes) 000876 1777421593.024757 mapwrite 0x028631 thru 0x02fc70 (0x7640 bytes) 000877 1777421593.052919 trunc from 0x0269d7 to 0x029141 (0x276b bytes) 000878 1777421593.056440 mapread 0x005139 thru 0x005e76 (0xd3e bytes) 000879 1777421593.057555 mapwrite 0x011c91 thru 0x0204ce (0xe83e bytes) 000881 1777421593.077392 read 0x021d4b thru 0x022087 (0x33d bytes) 000882 1777421593.078132 trunc from 0x02448d to 0x02c1d7 (0x7d4b bytes) 000883 1777421593.082073 trunc from 0x02fc71 to 0x031a54 (0x1de4 bytes) 000885 1777421593.092066 trunc from 0x00364c to 0x0061e5 (0x2b9a bytes) 000886 1777421593.098454 punch from 0x004be2 to 0x00b269 (0x6688 bytes) 000887 1777421593.101173 punch from 0x009454 to 0x013a39 (0xa5e6 bytes) 000888 1777421593.104977 trunc from 0x031a55 to 0x00ddf5 (0x23c5f bytes) 000889 1777421593.112073 fallocate 0x017f22 thru 0x018433 (0x512 bytes) 000890 1777421593.116363 write 0x011917 thru 0x017c79 (0x6363 bytes) 000891 1777421593.123701 read 0x012ba3 thru 0x018433 (0x5891 bytes) 000892 1777421593.124441 fallocate 0x0032f6 thru 0x008fe3 (0x5cee bytes) 000893 1777421593.127733 write 0x006d94 thru 0x015c99 (0xef06 bytes) 000894 1777421593.133119 trunc from 0x018434 to 0x012e21 (0x5612 bytes) 000895 1777421593.144389 read 0x00ce15 thru 0x012e21 (0x600d bytes) 000897 1777421593.145930 trunc from 0x012e22 to 0x03fc5c (0x2ce3b bytes) 000898 1777421593.157896 punch from 0x01651f to 0x0221aa (0xbc8c bytes) 000899 1777421593.164185 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fc5d): Mirror 17: 0x7f29fb5d (stale) Mirror 24: 0x7f29fb5d (stale) Mirror 25: 0x7f29fb5d (stale) Mirror 26: 0x7f29fb5d (stale) Mirror 27: 0x7f29fb5d (stale) Mirror 28: 0xb0f37555 (stale) Mirror 29: 0xb0f37555 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc5d): Mirror 17: 0x7f29fb5d (stale) Mirror 24: 0x7f29fb5d (stale) Mirror 25: 0x7f29fb5d (stale) Mirror 26: 0x7f29fb5d (stale) Mirror 27: 0x7f29fb5d (stale) Mirror 28: 0xb0f37555 (stale) Mirror 29: 0xb0f37555 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc5d: skipped 000900 1777421593.399773 write 0x000146 thru 0x0014d2 (0x138d bytes) 000901 1777421593.405633 write 0x00fc3e thru 0x01f7c8 (0xfb8b bytes) 000902 1777421593.410648 punch from 0x0161e1 to 0x019800 (0x3620 bytes) 000903 1777421593.418052 punch from 0x021ca1 to 0x02abc7 (0x8f27 bytes) 000906 1777421593.423516 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3fc5d): Mirror 17: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc5d): Mirror 17: 0x7f29fb5d (stale) Mirror 24: 0x7f29fb5d (stale) Mirror 25: 0x7f29fb5d (stale) Mirror 26: 0x7f29fb5d (stale) Mirror 27: 0x7f29fb5d (stale) Mirror 28: 0xb0f37555 (stale) Mirror 29: 0xb0f37555 (stale) CRC-32 checksum value for chunk [0, 0x3fc5d): Mirror 1: 0x33f7ab5a Mirror 30: 0x33f7ab5a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc5d: skipped 000907 1777421593.867972 mapwrite 0x01823d thru 0x021f89 (0x9d4d bytes) 000908 1777421593.929289 punch from 0x03944d to 0x03fc5c (0x6810 bytes) 000909 1777421593.932284 write 0x00f91c thru 0x0112a6 (0x198b bytes) 000910 1777421593.938585 mapwrite 0x01037e thru 0x018297 (0x7f1a bytes) 000911 1777421593.952548 trunc from 0x014ffe to 0x022f5e (0xdf61 bytes) 000912 1777421593.956933 trunc from 0x00ac71 to 0x014458 (0x97e8 bytes) 000913 1777421593.961417 mapwrite 0x016e47 thru 0x01b7b1 (0x496b bytes) 000914 1777421593.983937 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fc5d): Mirror 24: 0x7f29fb5d (stale) Mirror 25: 0x7f29fb5d (stale) Mirror 26: 0x7f29fb5d (stale) Mirror 27: 0x7f29fb5d (stale) Mirror 28: 0xb0f37555 (stale) Mirror 29: 0xb0f37555 (stale) Mirror 30: 0x33f7ab5a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc5d): Mirror 24: 0x7f29fb5d (stale) Mirror 25: 0x7f29fb5d (stale) Mirror 26: 0x7f29fb5d (stale) Mirror 27: 0x7f29fb5d (stale) Mirror 28: 0xb0f37555 (stale) Mirror 29: 0xb0f37555 (stale) Mirror 30: 0x33f7ab5a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc5d: skipped 000915 1777421594.227050 punch from 0x00c597 to 0x015e01 (0x986b bytes) 000916 1777421594.229684 mapread 0x008761 thru 0x0170bd (0xe95d bytes) 000917 1777421594.240043 punch from 0x026215 to 0x02ddc0 (0x7bac bytes) 000918 1777421594.244894 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fc5d): Mirror 25: 0x7f29fb5d (stale) Mirror 26: 0x7f29fb5d (stale) Mirror 27: 0x7f29fb5d (stale) Mirror 28: 0xb0f37555 (stale) Mirror 29: 0xb0f37555 (stale) Mirror 30: 0x33f7ab5a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc5d): Mirror 25: 0x7f29fb5d (stale) Mirror 26: 0x7f29fb5d (stale) Mirror 27: 0x7f29fb5d (stale) Mirror 28: 0xb0f37555 (stale) Mirror 29: 0xb0f37555 (stale) Mirror 30: 0x33f7ab5a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc5d: skipped 000920 1777421594.448556 mapread 0x03796f thru 0x03fc5c (0x82ee bytes) 000921 1777421594.460023 mapwrite 0x026794 thru 0x0352dd (0xeb4a bytes) 000922 1777421594.481091 write 0x01141a thru 0x013993 (0x257a bytes) 000923 1777421594.487588 mapwrite 0x009e32 thru 0x016d6c (0xcf3b bytes) 000925 1777421594.502575 write 0x0130a9 thru 0x0177b1 (0x4709 bytes) 000926 1777421594.503851 mapwrite 0x0186d8 thru 0x01ba9a (0x33c3 bytes) 000927 1777421594.516783 MIRROR_RESYNC 7 mirrors 000928 1777421595.292735 trunc from 0x02c037 to 0x02f01f (0x2fe9 bytes) 000929 1777421595.316981 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fc5d): Mirror 25: 0x97e218e2 (stale) Mirror 26: 0x97e218e2 (stale) Mirror 28: 0x97e218e2 (stale) Mirror 29: 0x97e218e2 (stale) Mirror 30: 0x97e218e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc5d): Mirror 25: 0x97e218e2 (stale) Mirror 26: 0x97e218e2 (stale) Mirror 28: 0x97e218e2 (stale) Mirror 29: 0x97e218e2 (stale) Mirror 30: 0x97e218e2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc5d: skipped 000930 1777421595.539679 punch from 0x03aab8 to 0x03fc5c (0x51a5 bytes) 000932 1777421595.545626 mapread 0x03dea1 thru 0x03fc5c (0x1dbc bytes) 000933 1777421595.561471 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3fc5d): Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc5d): Mirror 25: 0x97e218e2 (stale) Mirror 26: 0x97e218e2 (stale) Mirror 28: 0x97e218e2 (stale) Mirror 29: 0x97e218e2 (stale) Mirror 30: 0x97e218e2 (stale) CRC-32 checksum value for chunk [0, 0x3fc5d): Mirror 1: 0xb8a79d79 Mirror 31: 0xb8a79d79 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc5d: skipped 000934 1777421596.063610 trunc from 0x03fc5d to 0x009017 (0x36c45 bytes) 000935 1777421596.088829 mapread 0x006e24 thru 0x007974 (0xb51 bytes) 000936 1777421596.094974 fallocate 0x02dde0 thru 0x0392cd (0xb4ee bytes) 000937 1777421596.098059 read 0x000e5a thru 0x002730 (0x18d7 bytes) 000938 1777421596.104362 write 0x00f865 thru 0x019aba (0xa256 bytes) 000941 1777421596.107478 write 0x00e51a thru 0x011ac3 (0x35aa bytes) 000942 1777421596.112111 mapread 0x00c96d thru 0x00d34b (0x9df bytes) 000943 1777421596.113009 read 0x00a4db thru 0x011e10 (0x7936 bytes) 000944 1777421596.113543 trunc from 0x019abb to 0x01e042 (0x4588 bytes) 000945 1777421596.125789 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e043): Mirror 25: 0xebf9cd90 (stale) Mirror 26: 0xebf9cd90 (stale) Mirror 28: 0xebf9cd90 (stale) Mirror 29: 0xebf9cd90 (stale) Mirror 31: 0xebf9cd90 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e043): Mirror 25: 0xebf9cd90 (stale) Mirror 26: 0xebf9cd90 (stale) Mirror 28: 0xebf9cd90 (stale) Mirror 29: 0xebf9cd90 (stale) Mirror 31: 0xebf9cd90 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e043: skipped 000946 1777421596.303547 trunc from 0x0199ce to 0x01b67c (0x1caf bytes) 000947 1777421596.311106 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1e043): Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e043): Mirror 25: 0xebf9cd90 (stale) Mirror 26: 0xebf9cd90 (stale) Mirror 28: 0xebf9cd90 (stale) Mirror 29: 0xebf9cd90 (stale) Mirror 31: 0xebf9cd90 (stale) CRC-32 checksum value for chunk [0, 0x1e043): Mirror 1: 0x5de5e8ce Mirror 32: 0x5de5e8ce lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e043: skipped 000948 1777421596.645030 mapwrite 0x01372c thru 0x01852a (0x4dff bytes) 000949 1777421596.702057 trunc from 0x01e043 to 0x01f076 (0x1034 bytes) 000950 1777421596.727253 write 0x0231c1 thru 0x025301 (0x2141 bytes) 000951 1777421596.728322 read 0x020d3e thru 0x025301 (0x45c4 bytes) 000953 1777421596.732033 write 0x01c7b1 thru 0x01caa4 (0x2f4 bytes) 000954 1777421596.733355 punch from 0x004516 to 0x00f062 (0xab4d bytes) 000955 1777421596.736841 read 0x0141af thru 0x02061f (0xc471 bytes) 000956 1777421596.742976 mapwrite 0x00dce7 thru 0x019424 (0xb73e bytes) 000957 1777421596.759507 read 0x011215 thru 0x01a50b (0x92f7 bytes) 000959 1777421596.760615 fallocate 0x00664a thru 0x010686 (0xa03d bytes) 000960 1777421596.764179 trunc from 0x025302 to 0x00f023 (0x162de bytes) 000961 1777421596.772894 trunc from 0x00f024 to 0x014d1a (0x5cf7 bytes) 000963 1777421596.781759 trunc from 0x014d1b to 0x02fcc8 (0x1afae bytes) 000964 1777421596.787583 trunc from 0x010f8d to 0x0152ce (0x4342 bytes) 000965 1777421596.791368 trunc from 0x017df1 to 0x021f86 (0xa196 bytes) 000966 1777421596.794041 trunc from 0x00b522 to 0x016612 (0xb0f1 bytes) 000967 1777421596.799099 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fcc9): Mirror 26: 0xf2e664e7 (stale) Mirror 28: 0xf2e664e7 (stale) Mirror 29: 0xf2e664e7 (stale) Mirror 31: 0xa40af3f4 (stale) Mirror 32: 0xf89182e1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fcc9): Mirror 26: 0xf2e664e7 (stale) Mirror 28: 0xf2e664e7 (stale) Mirror 29: 0xf2e664e7 (stale) Mirror 31: 0xa40af3f4 (stale) Mirror 32: 0xf89182e1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fcc9: skipped 000968 1777421596.975380 trunc from 0x00db8c to 0x016e1c (0x9291 bytes) 000969 1777421596.979313 mapread 0x014ec0 thru 0x0249dd (0xfb1e bytes) 000970 1777421596.996002 punch from 0x02122e to 0x02b959 (0xa72c bytes) 000971 1777421597.000043 write 0x021fe5 thru 0x0308a6 (0xe8c2 bytes) 000972 1777421597.010362 write 0x02e6c5 thru 0x036cd5 (0x8611 bytes) 000973 1777421597.011726 trunc from 0x036cd6 to 0x017a36 (0x1f29f bytes) 000974 1777421597.022371 read 0x00226d thru 0x008379 (0x610d bytes) 000975 1777421597.026604 write 0x011c81 thru 0x01487b (0x2bfb bytes) 000976 1777421597.027549 mapwrite 0x0098e9 thru 0x012707 (0x8e1f bytes) 000977 1777421597.042614 punch from 0x00f69a to 0x01095b (0x12c2 bytes) 000978 1777421597.045514 trunc from 0x017a37 to 0x01b0c9 (0x3693 bytes) 000979 1777421597.053250 trunc from 0x01b0ca to 0x022e70 (0x7da7 bytes) 000980 1777421597.059890 read 0x021a62 thru 0x0220f1 (0x690 bytes) 000981 1777421597.065735 read 0x0195d4 thru 0x022936 (0x9363 bytes) 000982 1777421597.066384 fallocate 0x01d3d2 thru 0x024453 (0x7082 bytes) 000983 1777421597.070080 mapwrite 0x03c12e thru 0x03ed8d (0x2c60 bytes) 000984 1777421597.103657 trunc from 0x02e604 to 0x03cf5c (0xe959 bytes) 000985 1777421597.107064 punch from 0x0055fd to 0x007a1a (0x241e bytes) 000986 1777421597.109700 write 0x022587 thru 0x02e473 (0xbeed bytes) 000987 1777421597.116779 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x463:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ed8e): Mirror 28: 0x2b59e35e (stale) Mirror 29: 0x2b59e35e (stale) Mirror 31: 0x8e0eebec (stale) Mirror 32: 0x3dd010ba (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed8e): Mirror 28: 0x2b59e35e (stale) Mirror 29: 0x2b59e35e (stale) Mirror 31: 0x8e0eebec (stale) Mirror 32: 0x3dd010ba (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed8e: skipped 000988 1777421597.270631 trunc from 0x03ed8e to 0x017fc1 (0x26dcc bytes) 000989 1777421597.281338 write 0x033c76 thru 0x0352df (0x166a bytes) 000990 1777421597.282460 write 0x032f40 thru 0x03f3db (0xc49c bytes) 000991 1777421597.285687 punch from 0x0331f3 to 0x03d26e (0xa07c bytes) 000992 1777421597.294386 write 0x012372 thru 0x0186ba (0x6349 bytes) 000993 1777421597.303879 MIRROR_RESYNC 5 mirrors 000994 1777421597.844150 mapread 0x001e9a thru 0x007d6f (0x5ed6 bytes) 000996 1777421597.850460 mapread 0x021cce thru 0x02ea0a (0xcd3d bytes) 000997 1777421597.856904 trunc from 0x01312b to 0x01f25a (0xc130 bytes) 000998 1777421597.872527 mapwrite 0x02ef1a thru 0x037573 (0x865a bytes) 000999 1777421597.886554 write 0x03ed4f thru 0x03ffff (0x12b1 bytes) 001000 1777421597.887452 mapwrite 0x014779 thru 0x01a8c0 (0x6148 bytes) All operations completed A-OK! PASS 70a (55s) == sanity-flr test 71: check mirror extend parent fid ==== 20:13:19 (1777421599) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x55:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (4s) == sanity-flr test 72: check append on FLR file ========== 20:13:23 (1777421603) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x98:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0679834 s, 30.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x96:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x98:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506829 s, 20.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x96:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x99:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x98:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x96:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x99:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x98:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x97:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (3s) == sanity-flr test 200a: stress test ===================== 20:13:26 (1777421606) Starting client: oleg236-client.virtnet: -o user_xattr,flock 192.168.202.136@tcp:/lustre /mnt/lustre2 Starting client: oleg236-client.virtnet: -o user_xattr,flock 192.168.202.136@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4793056 .. Extending file size to 7235872 .. failed Extending file size to 5461472 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6440352 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8230080 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8323776 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7553760 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8329344 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8225344 .. Extending file size to 8550432 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9072736 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8722656 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 9195936 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9157664 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9327168 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61389 61390 61391 61393 61396 failed 192.168.202.136@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg236-client.virtnet /mnt/lustre2 (opts:) 192.168.202.136@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg236-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:14:32 (1777421672) Starting client: oleg236-client.virtnet: -o user_xattr,flock 192.168.202.136@tcp:/lustre /mnt/lustre2 Starting client: oleg236-client.virtnet: -o user_xattr,flock 192.168.202.136@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7641120 .. Extending file size to 3487808 .. done Extending file size to 5453600 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8220992 .. Extending file size to 5639808 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8444384 .. failed Extending file size to 8955296 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8997632 .. Extending file size to 9021856 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9204064 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 64666 64667 64669 64671 64672 192.168.202.136@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg236-client.virtnet /mnt/lustre2 (opts:) 192.168.202.136@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg236-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (66s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:15:38 (1777421738) Starting client: oleg236-client.virtnet: -o user_xattr,flock 192.168.202.136@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2572 1268732 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2688 1268616 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25876 3581144 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17876 3589144 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43752 7170288 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2572 1268732 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2688 1268616 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25876 3581144 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17876 3589144 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43752 7170288 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0475398 s, 44.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0992871 s, 21.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x66b88008 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.136@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg236-client.virtnet /mnt/lustre2 (opts:) PASS 200c (7s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:15:46 (1777421746) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0597476 s, 35.1 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 20:15:49 (1777421749) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x64:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:15:52 (1777421752) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0310336 s, 67.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x611:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x611:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x612:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:15:55 (1777421755) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0347977 s, 60.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:15:58 (1777421758) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0339392 s, 61.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x612:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x613:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x612:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x613:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x614:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:16:23 (1777421783) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg236-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373764 s, 28.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x613:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x616:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x615:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x613:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x616:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x615:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x614:0x0] } PASS 204d (39s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:17:02 (1777421822) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg236-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x617:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x618:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0412219 s, 50.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x617:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x618:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x615:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x617:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x618:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x615:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (40s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:17:42 (1777421862) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg236-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg236-server: Pool lustre.test_204f created Adding targets to pool oleg236-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg236-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0530126 s, 39.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x619:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x616:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x619:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x616:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg236-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg236-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg236-server: Pool lustre.test_204f destroyed PASS 204f (49s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:18:32 (1777421912) PASS 205a (4s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:18:35 (1777421915) PASS 205b (4s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:18:39 (1777421919) oleg236-server: Pool lustre.test_206 created Adding targets to pool oleg236-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg236-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg236-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg236-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg236-server: Pool lustre.test_206 destroyed PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:18:54 (1777421934) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028332 s, 37.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191696 s, 54.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xe0c6a827 Mirror 3: 0xe0c6a827 PASS 207 (4s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:18:58 (1777421938) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:18:59 (1777421939) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:19:01 (1777421941) Starting client: oleg236-client.virtnet: -o user_xattr,flock 192.168.202.136@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0380404 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1670000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x24e0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x24e0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x24e0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2380000 Read this (4 bytes): BBBB 192.168.202.136@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg236-client.virtnet /mnt/lustre2 (opts:) PASS 209a (19s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:19:20 (1777421960) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0207767 s, 394 kB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:19:23 (1777421963) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199998 s, 52.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7f lmm_fid: [0x200000405:0x7f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7f lmm_fid: [0x200000405:0x7f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbd:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611263 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-28 20:19:24.000000000 -0400 Modify: 2026-04-28 20:19:24.000000000 -0400 Change: 2026-04-28 20:19:25.000000000 -0400 Birth: 2026-04-28 20:19:24.000000000 -0400 PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:19:27 (1777421967) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131233 s, 79.9 MB/s oleg236-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg236-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg236-server: O/280000401/d24/b8: File not found by ext2_lookup PASS 210b (13s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:19:40 (1777421980) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.112913 s, 363 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0193082 s, 212 kB/s size after second write -rw-r--r-- 1 root root 4096 Apr 28 20:19 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x84:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 28 20:19 /mnt/lustre/f211.sanity-flr PASS 211 (4s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:19:44 (1777421984) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (4s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:19:48 (1777421988) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (3s) == sanity-flr test complete, duration 914 sec ============ 20:19:51 (1777421991) === sanity-flr: start cleanup 20:19:52 (1777421992) === === sanity-flr: finish cleanup 20:19:55 (1777421995) ===