-----============= acceptance-small: sanity-flr ============----- Fri Apr 24 11:23:22 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg126-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 11:24:22 (1777044262) === oleg126-client.virtnet: executing check_config_client /mnt/lustre oleg126-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg126-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9cfc98186000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9cfc98186000.idle_timeout=debug disable quota as required oleg126-server: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 11:26:05 (1777044365) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7647] == sanity-flr test 0a: lfs mirror create with -N option == 11:26:13 (1777044373) PASS 0a (61s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 11:27:14 (1777044434) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (11s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 11:27:26 (1777044446) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (9s) == sanity-flr test 0d: lfs mirror extend with -N option == 11:27:36 (1777044456) PASS 0d (90s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 11:29:05 (1777044545) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (10s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 11:29:16 (1777044556) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (10s) == sanity-flr test 0g: lfs mirror create flags support === 11:29:27 (1777044567) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9cfc98186000.stats=clear osc.lustre-OST0001-osc-ffff9cfc98186000.stats=clear PASS 0g (25s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 11:29:53 (1777044593) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (40s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 11:30:34 (1777044634) PASS 0j (27s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 11:31:02 (1777044662) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (10s) == sanity-flr test 0m: mirror extend with -N= option ===== 11:31:12 (1777044672) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (24s) == sanity-flr test 1: create components with setstripe options ========================================================== 11:31:37 (1777044697) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (36s) == sanity-flr test 2: create components from existing files ========================================================== 11:32:13 (1777044733) PASS 2 (23s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 11:32:37 (1777044757) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (12s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 11:32:48 (1777044768) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (24s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 11:33:13 (1777044793) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (23s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 11:33:47 (1777044827) PASS 7 (22s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 11:34:10 (1777044850) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.880173 s, 1.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.754424 s, 1.4 MB/s PASS 21 (29s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 11:34:39 (1777044879) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.84582 s, 9.1 MB/s PASS 22 (31s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 11:35:10 (1777044910) Stopping /mnt/lustre-ost1 (opts:) on oleg126-server oleg126-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9cfc98186000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9cfc98186000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg126-server: oleg126-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg126-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg126-server oleg126-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9cfc98186000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9cfc98186000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg126-server: oleg126-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg126-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (229s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 11:38:59 (1777045139) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 3.56562 s, 3.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg126-server oleg126-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9cfc98186000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9cfc98186000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg126-server: oleg126-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg126-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (158s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 11:41:38 (1777045298) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg126-server oleg126-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9cfc98186000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9cfc98186000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg126-server: oleg126-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg126-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg126-server oleg126-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9cfc98186000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9cfc98186000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg126-server: oleg126-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg126-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (236s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 11:45:34 (1777045534) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 28.9207 s, 1.8 MB/s -rw-r--r-- 1 root root 52428800 Apr 24 11:46 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m7.553s user 0m0.000s sys 0m0.333s Stopping /mnt/lustre-ost1 (opts:) on oleg126-server oleg126-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9cfc98186000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9cfc98186000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m5.400s user 0m0.003s sys 0m0.234s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg126-server: oleg126-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg126-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg126-server oleg126-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9cfc98186000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9cfc98186000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m4.648s user 0m0.000s sys 0m0.209s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg126-server: oleg126-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg126-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg126-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9cfc98186000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9cfc98186000.ost_server_uuid in FULL state after 0 sec PASS 33b (290s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 11:50:23 (1777045823) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (8s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 11:50:32 (1777045832) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (9s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 11:50:40 (1777045840) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (8s) == sanity-flr test 35: allow to write to mirrored files == 11:50:49 (1777045849) PASS 35 (21s) == sanity-flr test 36a: write to mirrored files ========== 11:51:11 (1777045871) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 1.80275 s, 7.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (36s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 11:51:48 (1777045908) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206229 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208585 s, 5.0 MB/s PASS 36b (22s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 11:52:10 (1777045930) PASS 36c (22s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 11:52:32 (1777045952) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00226267 s, 453 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00453337 s, 226 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00868376 s, 118 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00151464 s, 676 kB/s flv=10 olv=10 PASS 36d (40s) == sanity-flr test 37: mirror I/O API verification ======= 11:53:13 (1777045993) create test files with size 15754589 .. 6ecaf7d1365ca8bb97d5be022d234b46 - d4e797457fc213230bb5ec9a84c2f032 - 3c046c1b2ef9b53fb8b61bffee3a211b - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg126-client.virtnet /mnt/lustre (opts:) Starting client: oleg126-client.virtnet: -o user_xattr,flock 192.168.201.126@tcp:/lustre /mnt/lustre PASS 37 (82s) == sanity-flr test 38: resync ============================ 11:54:35 (1777046075) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (42s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 11:55:18 (1777046118) PASS 39 (20s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 11:55:38 (1777046138) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.5642 s, 1.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170399 s, 6.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (33s) == sanity-flr test 41: lfs mirror resync check =========== 11:56:11 (1777046171) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.676237 s, 3.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 1.50541 s, 2.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.932107 s, 2.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (54s) == sanity-flr test 42: lfs mirror verify ================= 11:57:05 (1777046225) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (11s) == sanity-flr test 43a: mirror pick on write ============= 11:57:16 (1777046236) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (10s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 11:57:27 (1777046247) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210088 s, 5.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (28s) == sanity-flr test 44a: lfs mirror split check =========== 11:57:56 (1777046276) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 11:58:04 (1777046284) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4e:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (27s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 11:58:36 (1777046316) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.872568 s, 12.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg126-client.virtnet /mnt/lustre (opts:) Starting client: oleg126-client.virtnet: -o user_xattr,flock 192.168.201.126@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (39s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 11:59:15 (1777046355) llite.lustre-ffff9cfc84dc9000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 1.31346 s, 8.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 Stopping client oleg126-client.virtnet /mnt/lustre (opts:) Starting client: oleg126-client.virtnet: -o user_xattr,flock 192.168.201.126@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 llite.lustre-ffff9cfc88f12000.xattr_cache=1 PASS 44e (46s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:00:02 (1777046402) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (116s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:01:57 (1777046517) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (118s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:03:56 (1777046636) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (11s) == sanity-flr test 48: Verify snapshot mirror ============ 12:04:08 (1777046648) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.8185 s, 3.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.692996 s, 4.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x757d7a39 (stale) Mirror 2: 0x757d7a39 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: f245fe623a7be0e94349da56b0f365ab - mirror 1 checksum: 45bbc09e3721a8ddb59098fa76b11d9e - mirror 2 checksum: f245fe623a7be0e94349da56b0f365ab - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Apr 24 12:04:20 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Apr 24 12:04:20 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: f245fe623a7be0e94349da56b0f365ab - mirror 1 checksum: 45bbc09e3721a8ddb59098fa76b11d9e - mirror 2 checksum: 45bbc09e3721a8ddb59098fa76b11d9e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Apr 24 12:04:20 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Apr 24 12:04:20 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } PASS 48 (38s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:04:51 (1777046691) Failing mds1 on oleg126-server Stopping /mnt/lustre-mds1 (opts:) on oleg126-server 12:05:14 (1777046714) shut down facet: mds1 facet_host: oleg126-server facet_failover_host: oleg126-server Failover mds1 to oleg126-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg126-server: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-MDT0000 12:06:15 (1777046775) targets are mounted 12:06:15 (1777046775) facet_failover done oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872481 s, 12.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 50A (126s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:06:58 (1777046818) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:07:08 (1777046828) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:07:16 (1777046836) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0550358 s, 298 kB/s PASS 50c (23s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:07:39 (1777046859) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:07:48 (1777046868) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:07:59 (1777046879) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (70s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:09:09 (1777046949) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (57s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:10:06 (1777047006) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (42s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:10:48 (1777047048) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (40s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:11:29 (1777047089) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (20s) == sanity-flr test 70: mirror create and split race ====== 12:11:50 (1777047110) mirror create pid 53268 mirror split pid 53269 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4993: 53268 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4993: 53269 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (79s) == sanity-flr test 70a: flr mode fsx test ================ 12:13:10 (1777047190) Seed set to 7194 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000004 1777047195.288714 write 0x0007ba thru 0x005730 (0x4f77 bytes) truncating to largest ever: 0x33d64 000005 1777047195.345298 trunc from 0x005731 to 0x033d63 (0x2e633 bytes) 000007 1777047195.418452 trunc from 0x033d64 to 0x02c8ff (0x7464 bytes) 000008 1777047195.485051 mapwrite 0x01243f thru 0x0128d9 (0x49b bytes) 000009 1777047195.808672 read 0x0183bb thru 0x01c406 (0x404c bytes) 000012 1777047195.866346 trunc from 0x02c900 to 0x033761 (0x6e62 bytes) 000016 1777047195.958776 read 0x009599 thru 0x00ea95 (0x54fd bytes) 000017 1777047196.006329 write 0x018e91 thru 0x01aa00 (0x1b70 bytes) 000018 1777047196.019519 trunc from 0x033762 to 0x02de85 (0x58dc bytes) 000019 1777047196.085304 read 0x011ab9 thru 0x01eb77 (0xd0bf bytes) 000021 1777047196.093608 write 0x010209 thru 0x012cb8 (0x2ab0 bytes) 000023 1777047196.115292 mapwrite 0x0366ff thru 0x03ffff (0x9901 bytes) 000025 1777047196.535854 read 0x018f16 thru 0x01bd5b (0x2e46 bytes) 000028 1777047196.557519 trunc from 0x040000 to 0x00f3f3 (0x30c0c bytes) 000029 1777047196.692079 MIRROR_EXTEND to 2 mirrors 000030 1777047199.242614 write 0x03b462 thru 0x03ffff (0x4b9e bytes) 000031 1777047199.526948 read 0x01b404 thru 0x022c42 (0x783f bytes) 000032 1777047199.571907 read 0x01003a thru 0x017797 (0x775e bytes) 000033 1777047199.634289 write 0x018b79 thru 0x025f55 (0xd3dd bytes) 000035 1777047199.693292 MIRROR_RESYNC 2 mirrors 000036 1777047202.310600 write 0x005a7f thru 0x00cd92 (0x7314 bytes) 000037 1777047202.618846 trunc from 0x040000 to 0x023266 (0x1cd99 bytes) 000039 1777047202.720442 mapread 0x003d65 thru 0x011b49 (0xdde5 bytes) 000040 1777047202.797841 write 0x0062c4 thru 0x009099 (0x2dd6 bytes) 000044 1777047202.845697 mapread 0x0155f1 thru 0x0186cd (0x30dd bytes) 000045 1777047202.881094 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xba:0x0]: Invalid argument (22) 000046 1777047203.551816 write 0x013775 thru 0x013e4e (0x6da bytes) 000050 1777047203.606567 write 0x027941 thru 0x027bae (0x26e bytes) 000053 1777047203.615602 mapread 0x00439b thru 0x005eb0 (0x1b16 bytes) 000056 1777047203.759276 read 0x009a48 thru 0x00a54c (0xb05 bytes) 000057 1777047203.760685 mapwrite 0x028a3c thru 0x02fb16 (0x70db bytes) 000059 1777047204.353879 mapread 0x021e56 thru 0x02d951 (0xbafc bytes) 000060 1777047204.370193 read 0x02e84e thru 0x02fb16 (0x12c9 bytes) 000061 1777047204.388282 read 0x01c25b thru 0x028810 (0xc5b6 bytes) 000062 1777047204.450248 mapwrite 0x035bf2 thru 0x037ad6 (0x1ee5 bytes) 000064 1777047205.015783 trunc from 0x037ad7 to 0x0157eb (0x222eb bytes) 000065 1777047205.134556 read 0x000af4 thru 0x002dd7 (0x22e4 bytes) 000068 1777047205.147962 read 0x0032e1 thru 0x00b564 (0x8284 bytes) 000071 1777047205.160989 mapread 0x01271c thru 0x0157eb (0x30d0 bytes) 000072 1777047205.165178 write 0x024332 thru 0x02bbbb (0x788a bytes) 000074 1777047205.173340 read 0x025eb7 thru 0x026083 (0x1cd bytes) 000075 1777047205.176534 read 0x0188c3 thru 0x023350 (0xaa8e bytes) 000076 1777047205.214016 trunc from 0x02bbbc to 0x00f600 (0x1c5bb bytes) 000080 1777047205.315426 read 0x0004f0 thru 0x001e1f (0x1930 bytes) 000082 1777047205.344250 write 0x00092d thru 0x001d92 (0x1466 bytes) 000083 1777047205.353998 write 0x000562 thru 0x0041c6 (0x3c65 bytes) 000085 1777047205.389713 MIRROR_EXTEND to 2 mirrors 000086 1777047208.280240 mapread 0x00628c thru 0x00f600 (0x9375 bytes) 000087 1777047208.536632 write 0x02980b thru 0x038de7 (0xf5dd bytes) 000088 1777047209.031872 read 0x017551 thru 0x01a139 (0x2be9 bytes) 000089 1777047209.132934 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xba:0x0]: Invalid argument (22) 000090 1777047210.017433 mapwrite 0x018fff thru 0x020d9e (0x7da0 bytes) 000091 1777047210.738291 mapwrite 0x03cbd8 thru 0x03ffff (0x3428 bytes) 000093 1777047211.165152 mapwrite 0x020be2 thru 0x0306c8 (0xfae7 bytes) 000094 1777047211.744794 write 0x03672b thru 0x038477 (0x1d4d bytes) 000096 1777047211.821067 MIRROR_EXTEND to 2 mirrors 000097 1777047215.384631 MIRROR_RESYNC 2 mirrors 000102 1777047217.101495 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xba:0x0]: Invalid argument (22) 000103 1777047217.995622 mapread 0x02afb7 thru 0x0346a7 (0x96f1 bytes) 000104 1777047218.144639 read 0x02d2d7 thru 0x0372e2 (0xa00c bytes) 000105 1777047218.167825 trunc from 0x040000 to 0x017e54 (0x281ab bytes) 000106 1777047218.243301 mapread 0x016429 thru 0x017e54 (0x1a2c bytes) 000109 1777047218.366780 write 0x018769 thru 0x022f65 (0xa7fd bytes) 000110 1777047218.391536 mapwrite 0x01ced3 thru 0x023a6a (0x6b98 bytes) truncating to largest ever: 0x3a194 000111 1777047218.849995 trunc from 0x023a6b to 0x03a193 (0x16729 bytes) 000113 1777047218.961251 trunc from 0x03a194 to 0x024097 (0x160fc bytes) 000114 1777047219.063037 mapread 0x01d3c0 thru 0x024097 (0x6cd8 bytes) 000117 1777047219.126677 write 0x0250a0 thru 0x02d161 (0x80c2 bytes) 000118 1777047219.147486 mapwrite 0x0097d4 thru 0x00e895 (0x50c2 bytes) 000119 1777047219.451524 read 0x01e42d thru 0x02d161 (0xed35 bytes) truncating to largest ever: 0x3fd74 000122 1777047219.471146 trunc from 0x02d162 to 0x03fd73 (0x12c12 bytes) 000124 1777047219.608000 read 0x032834 thru 0x03ae64 (0x8631 bytes) 000125 1777047219.712598 MIRROR_EXTEND to 2 mirrors 000126 1777047223.932894 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xba:0x0]: Invalid argument (22) 000127 1777047224.725771 write 0x021ef6 thru 0x02f689 (0xd794 bytes) 000128 1777047224.817688 mapwrite 0x03437b thru 0x03ffff (0xbc85 bytes) 000129 1777047225.300148 read 0x00c608 thru 0x00f77e (0x3177 bytes) 000130 1777047225.333309 trunc from 0x040000 to 0x0005d3 (0x3fa2c bytes) 000133 1777047225.412724 trunc from 0x0005d4 to 0x01b1b7 (0x1abe4 bytes) 000134 1777047225.531415 read 0x019314 thru 0x01b1b7 (0x1ea4 bytes) 000135 1777047225.580379 mapread 0x0154bb thru 0x01b1b7 (0x5cfd bytes) 000136 1777047225.603572 mapread 0x019dd3 thru 0x01b1b7 (0x13e5 bytes) 000140 1777047225.625159 read 0x008bfe thru 0x00fa59 (0x6e5c bytes) 000141 1777047225.628230 read 0x011378 thru 0x01b1b7 (0x9e40 bytes) 000143 1777047225.644682 mapread 0x007b2a thru 0x00ae17 (0x32ee bytes) 000145 1777047225.666198 MIRROR_EXTEND to 2 mirrors 000146 1777047228.330973 write 0x000daf thru 0x00133d (0x58f bytes) 000149 1777047228.685201 read 0x006f2c thru 0x011067 (0xa13c bytes) 000151 1777047228.794254 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xba:0x0]: Invalid argument (22) 000152 1777047229.541813 read 0x01a388 thru 0x01b1b7 (0xe30 bytes) 000154 1777047229.598511 mapwrite 0x02f9df thru 0x03bd8c (0xc3ae bytes) 000155 1777047230.387108 mapread 0x015067 thru 0x022a88 (0xda22 bytes) 000158 1777047230.586855 MIRROR_EXTEND to 2 mirrors 000159 1777047234.434506 write 0x007290 thru 0x00c933 (0x56a4 bytes) 000160 1777047234.876881 mapread 0x006fb7 thru 0x009877 (0x28c1 bytes) 000161 1777047234.970559 mapwrite 0x03f16a thru 0x03ffff (0xe96 bytes) 000162 1777047235.425971 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xea4cab3e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd86580ca Mirror 3: 0xd86580ca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000164 1777047240.145807 mapread 0x01a3ed thru 0x022e30 (0x8a44 bytes) 000165 1777047240.338718 mapwrite 0x0152bf thru 0x01e656 (0x9398 bytes) 000166 1777047241.150043 MIRROR_RESYNC 3 mirrors 000167 1777047244.774167 write 0x0135df thru 0x01f3e2 (0xbe04 bytes) 000168 1777047245.056195 read 0x02e426 thru 0x03cd0a (0xe8e5 bytes) 000173 1777047245.187434 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6ff42be4 (stale) Mirror 3: 0x6ff42be4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb43d1333 Mirror 4: 0xb43d1333 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000174 1777047248.905727 read 0x0348a6 thru 0x03f8f0 (0xb04b bytes) 000176 1777047248.984323 mapwrite 0x0355cb thru 0x03ad6a (0x57a0 bytes) 000178 1777047250.066265 write 0x02f2f6 thru 0x032414 (0x311f bytes) 000179 1777047250.116750 write 0x023d29 thru 0x02d55f (0x9837 bytes) 000182 1777047250.207689 mapwrite 0x0399d7 thru 0x03ffff (0x6629 bytes) 000188 1777047250.732744 mapread 0x02072d thru 0x023699 (0x2f6d bytes) 000189 1777047250.758114 mapread 0x02124f thru 0x0281bf (0x6f71 bytes) 000191 1777047250.768316 trunc from 0x040000 to 0x03bd68 (0x4297 bytes) 000192 1777047250.871212 mapread 0x014fd3 thru 0x019e44 (0x4e72 bytes) 000193 1777047251.027261 MIRROR_RESYNC 4 mirrors 000194 1777047255.145530 mapread 0x01507b thru 0x0183b6 (0x333c bytes) 000195 1777047255.330976 MIRROR_EXTEND to 5 mirrors 000196 1777047258.222712 read 0x036e53 thru 0x03bd68 (0x4f16 bytes) 000200 1777047258.266089 mapwrite 0x01adea thru 0x0271c0 (0xc3d7 bytes) 000201 1777047259.029788 trunc from 0x03bd69 to 0x00ed57 (0x2d011 bytes) 000203 1777047259.140839 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xed58): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xed58): Mirror 2: 0xf1378191 (stale) Mirror 3: 0xf1378191 (stale) Mirror 4: 0xf1378191 (stale) Mirror 5: 0xf1378191 (stale) CRC-32 checksum value for chunk [0, 0xed58): Mirror 1: 0xf1378191 Mirror 6: 0xf1378191 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xed58: skipped 000204 1777047262.983008 trunc from 0x00ed58 to 0x036831 (0x27ada bytes) 000206 1777047263.467759 MIRROR_RESYNC 6 mirrors 000208 1777047268.445387 mapwrite 0x02aecc thru 0x0302b1 (0x53e6 bytes) 000210 1777047269.405274 mapread 0x028d4c thru 0x031861 (0x8b16 bytes) 000211 1777047269.472544 MIRROR_RESYNC 6 mirrors 000212 1777047274.447975 mapwrite 0x005e5b thru 0x009213 (0x33b9 bytes) 000213 1777047275.408987 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x36832): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36832): Mirror 2: 0x2cab4572 (stale) Mirror 3: 0x2cab4572 (stale) Mirror 4: 0x2cab4572 (stale) Mirror 5: 0x2cab4572 (stale) Mirror 6: 0x2cab4572 (stale) CRC-32 checksum value for chunk [0, 0x36832): Mirror 1: 0x69a0ec14 Mirror 7: 0x69a0ec14 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36832: skipped 000214 1777047279.937941 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36832): Mirror 2: 0x2cab4572 (stale) Mirror 3: 0x2cab4572 (stale) Mirror 5: 0x2cab4572 (stale) Mirror 6: 0x2cab4572 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36832): Mirror 2: 0x2cab4572 (stale) Mirror 3: 0x2cab4572 (stale) Mirror 5: 0x2cab4572 (stale) Mirror 6: 0x2cab4572 (stale) CRC-32 checksum value for chunk [0, 0x36832): Mirror 1: 0x69a0ec14 Mirror 7: 0x69a0ec14 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36832: skipped 000216 1777047283.362558 read 0x0049c6 thru 0x00579a (0xdd5 bytes) 000219 1777047283.688659 trunc from 0x036832 to 0x002cc7 (0x33b6a bytes) 000220 1777047284.114764 mapread 0x002098 thru 0x002cc7 (0xc30 bytes) 000221 1777047284.146517 mapread 0x00231f thru 0x002cc7 (0x9a9 bytes) 000225 1777047284.178199 read 0x000195 thru 0x002cc7 (0x2b33 bytes) 000226 1777047284.254725 trunc from 0x002cc8 to 0x005b31 (0x2e6a bytes) 000227 1777047284.351154 read 0x000334 thru 0x0029e5 (0x26b2 bytes) 000228 1777047284.360771 mapwrite 0x012536 thru 0x01b301 (0x8dcc bytes) 000233 1777047284.941076 trunc from 0x01b302 to 0x020089 (0x4d88 bytes) 000234 1777047285.023670 mapread 0x010385 thru 0x01bab4 (0xb730 bytes) 000235 1777047285.041679 mapread 0x0170ba thru 0x020089 (0x8fd0 bytes) 000236 1777047285.108257 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2008a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2008a): Mirror 2: 0xfdc51f1b (stale) Mirror 3: 0xfdc51f1b (stale) Mirror 5: 0xfdc51f1b (stale) Mirror 6: 0xfdc51f1b (stale) Mirror 7: 0x6e1ac809 (stale) CRC-32 checksum value for chunk [0, 0x2008a): Mirror 1: 0xd5e9a42c Mirror 8: 0xd5e9a42c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2008a: skipped 000238 1777047288.923938 read 0x01954c thru 0x01f1b1 (0x5c66 bytes) 000239 1777047288.967355 read 0x01d956 thru 0x020089 (0x2734 bytes) 000241 1777047289.069140 mapread 0x00cc66 thru 0x00e709 (0x1aa4 bytes) 000242 1777047289.133658 mapread 0x004d4c thru 0x0075a0 (0x2855 bytes) 000244 1777047289.249282 mapread 0x01bb8c thru 0x020089 (0x44fe bytes) 000245 1777047289.325646 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2008a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2008a): Mirror 2: 0xfdc51f1b (stale) Mirror 3: 0xfdc51f1b (stale) Mirror 5: 0xfdc51f1b (stale) Mirror 6: 0xfdc51f1b (stale) Mirror 7: 0x6e1ac809 (stale) CRC-32 checksum value for chunk [0, 0x2008a): Mirror 1: 0xd5e9a42c Mirror 8: 0xd5e9a42c Mirror 9: 0xd5e9a42c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2008a: skipped 000247 1777047293.443908 trunc from 0x02008a to 0x02d581 (0xd4f8 bytes) 000248 1777047293.879196 mapread 0x00b494 thru 0x0123f6 (0x6f63 bytes) 000249 1777047294.010010 trunc from 0x02d582 to 0x03a08c (0xcb0b bytes) 000250 1777047294.088721 write 0x028440 thru 0x02bc58 (0x3819 bytes) 000251 1777047294.149866 trunc from 0x03a08d to 0x01b7d6 (0x1e8b6 bytes) 000253 1777047294.261068 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b7d7): Mirror 2: 0x40c2d3b9 (stale) Mirror 3: 0x40c2d3b9 (stale) Mirror 5: 0x40c2d3b9 (stale) Mirror 6: 0x40c2d3b9 (stale) Mirror 7: 0x8f887458 (stale) Mirror 8: 0xab68ebb8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b7d7): Mirror 2: 0x40c2d3b9 (stale) Mirror 3: 0x40c2d3b9 (stale) Mirror 5: 0x40c2d3b9 (stale) Mirror 6: 0x40c2d3b9 (stale) Mirror 7: 0x8f887458 (stale) Mirror 8: 0xab68ebb8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b7d7: skipped 000259 1777047296.514918 write 0x0320d6 thru 0x03ffff (0xdf2a bytes) 000260 1777047296.542679 write 0x0076c5 thru 0x008008 (0x944 bytes) 000262 1777047296.586761 mapread 0x022f2c thru 0x02db9b (0xac70 bytes) 000263 1777047296.649637 trunc from 0x040000 to 0x02244e (0x1dbb1 bytes) 000265 1777047296.722688 trunc from 0x02244f to 0x035cfa (0x138ac bytes) 000267 1777047296.818311 mapread 0x025cec thru 0x027dd0 (0x20e5 bytes) 000268 1777047296.860712 mapread 0x001525 thru 0x005f5d (0x4a39 bytes) 000269 1777047296.918802 read 0x020ab6 thru 0x022374 (0x18bf bytes) 000270 1777047296.926888 write 0x004d17 thru 0x007a3b (0x2d25 bytes) 000271 1777047296.943310 read 0x033cd9 thru 0x034477 (0x79f bytes) 000274 1777047297.001803 read 0x00c48f thru 0x017c80 (0xb7f2 bytes) 000276 1777047297.006587 mapread 0x007f65 thru 0x016e1e (0xeeba bytes) 000278 1777047297.016894 write 0x01ae62 thru 0x01c55f (0x16fe bytes) 000279 1777047297.018902 mapread 0x007363 thru 0x00c4e1 (0x517f bytes) 000280 1777047297.026031 mapwrite 0x03e0ee thru 0x03ffff (0x1f12 bytes) 000281 1777047297.442282 write 0x0242ba thru 0x02b35e (0x70a5 bytes) 000282 1777047297.455437 write 0x01a768 thru 0x023a74 (0x930d bytes) 000285 1777047297.484955 MIRROR_RESYNC 7 mirrors 000286 1777047303.262923 trunc from 0x040000 to 0x00fd72 (0x3028d bytes) 000288 1777047303.598368 mapread 0x00c919 thru 0x00f331 (0x2a19 bytes) 000289 1777047303.662505 mapwrite 0x027ebb thru 0x035386 (0xd4cc bytes) 000290 1777047304.173905 trunc from 0x035387 to 0x0056c8 (0x2fcbe bytes) 000291 1777047304.262126 trunc from 0x0056c9 to 0x010598 (0xaed0 bytes) 000292 1777047304.370771 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x10599): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10599): Mirror 2: 0x714c3bad (stale) Mirror 3: 0x714c3bad (stale) Mirror 5: 0x714c3bad (stale) Mirror 6: 0x714c3bad (stale) Mirror 7: 0x714c3bad (stale) Mirror 8: 0x714c3bad (stale) CRC-32 checksum value for chunk [0, 0x10599): Mirror 1: 0xbefe8f0a Mirror 9: 0xbefe8f0a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10599: skipped 000294 1777047307.818425 mapread 0x00a01d thru 0x010598 (0x657c bytes) 000295 1777047308.033162 mapread 0x005d6f thru 0x010598 (0xa82a bytes) 000296 1777047308.170841 read 0x00117f thru 0x00c8fa (0xb77c bytes) 000298 1777047308.284657 mapread 0x000508 thru 0x010149 (0xfc42 bytes) 000299 1777047308.421772 trunc from 0x010599 to 0x020c09 (0x10671 bytes) 000300 1777047308.874827 write 0x01b119 thru 0x02298d (0x7875 bytes) 000301 1777047308.922814 write 0x01376d thru 0x021d39 (0xe5cd bytes) 000302 1777047308.998588 MIRROR_RESYNC 8 mirrors 000303 1777047315.230516 write 0x00d8d8 thru 0x019dfb (0xc524 bytes) 000304 1777047315.768742 read 0x0162b2 thru 0x01ce7f (0x6bce bytes) 000305 1777047315.788637 mapwrite 0x03a2bd thru 0x03ffff (0x5d43 bytes) 000306 1777047316.220500 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xaa670d41 (stale) Mirror 3: 0xaa670d41 (stale) Mirror 5: 0xaa670d41 (stale) Mirror 6: 0xaa670d41 (stale) Mirror 7: 0xaa670d41 (stale) Mirror 8: 0xaa670d41 (stale) Mirror 9: 0xaa670d41 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2c31bfdf Mirror 10: 0x2c31bfdf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000310 1777047320.481955 mapread 0x002d80 thru 0x003568 (0x7e9 bytes) 000311 1777047320.654781 mapread 0x00da45 thru 0x013b65 (0x6121 bytes) 000312 1777047320.799499 trunc from 0x040000 to 0x03f048 (0xfb7 bytes) 000313 1777047321.331327 write 0x0180c5 thru 0x01996d (0x18a9 bytes) 000315 1777047321.453049 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3f049): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f049): Mirror 2: 0x6d0b3445 (stale) Mirror 3: 0x6d0b3445 (stale) Mirror 5: 0x6d0b3445 (stale) Mirror 6: 0x6d0b3445 (stale) Mirror 7: 0x6d0b3445 (stale) Mirror 8: 0x6d0b3445 (stale) Mirror 9: 0x6d0b3445 (stale) Mirror 10: 0xe8e2cdd3 (stale) CRC-32 checksum value for chunk [0, 0x3f049): Mirror 1: 0x8df0fdcb Mirror 11: 0x8df0fdcb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f049: skipped 000320 1777047325.861337 mapread 0x03015a thru 0x03230e (0x21b5 bytes) 000321 1777047326.021159 write 0x03997d thru 0x03ffff (0x6683 bytes) 000323 1777047326.479090 trunc from 0x040000 to 0x00bd2f (0x342d0 bytes) 000326 1777047326.708597 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0xbd30): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbd30): Mirror 2: 0x7fd943ac (stale) Mirror 3: 0x7fd943ac (stale) Mirror 5: 0x7fd943ac (stale) Mirror 6: 0x7fd943ac (stale) Mirror 7: 0x7fd943ac (stale) Mirror 8: 0x7fd943ac (stale) Mirror 9: 0x7fd943ac (stale) Mirror 10: 0x7fd943ac (stale) Mirror 11: 0x7fd943ac (stale) CRC-32 checksum value for chunk [0, 0xbd30): Mirror 1: 0x7fd943ac Mirror 12: 0x7fd943ac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbd30: skipped 000328 1777047330.854801 write 0x00095a thru 0x00339c (0x2a43 bytes) 000330 1777047331.345460 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0xbd30): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbd30): Mirror 2: 0x7fd943ac (stale) Mirror 3: 0x7fd943ac (stale) Mirror 5: 0x7fd943ac (stale) Mirror 6: 0x7fd943ac (stale) Mirror 7: 0x7fd943ac (stale) Mirror 8: 0x7fd943ac (stale) Mirror 9: 0x7fd943ac (stale) Mirror 10: 0x7fd943ac (stale) Mirror 11: 0x7fd943ac (stale) Mirror 12: 0x7fd943ac (stale) CRC-32 checksum value for chunk [0, 0xbd30): Mirror 1: 0x449abaa5 Mirror 13: 0x449abaa5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbd30: skipped 000332 1777047335.016569 write 0x014d7b thru 0x01f654 (0xa8da bytes) 000333 1777047335.435584 mapwrite 0x00a326 thru 0x017c8f (0xd96a bytes) 000335 1777047335.867293 read 0x00d2b1 thru 0x0144b3 (0x7203 bytes) 000338 1777047335.883127 mapwrite 0x03c9a2 thru 0x03d6b2 (0xd11 bytes) 000339 1777047336.302341 mapread 0x010945 thru 0x01ac4a (0xa306 bytes) 000340 1777047336.321463 mapwrite 0x03981a thru 0x03ffff (0x67e6 bytes) 000342 1777047336.725954 MIRROR_RESYNC 12 mirrors 000343 1777047344.215413 read 0x008c5b thru 0x00cf98 (0x433e bytes) 000344 1777047344.289799 read 0x02b6b7 thru 0x039f61 (0xe8ab bytes) 000348 1777047344.370222 read 0x004490 thru 0x0076b9 (0x322a bytes) 000349 1777047344.373164 mapwrite 0x03114c thru 0x035375 (0x422a bytes) 000350 1777047345.149227 mapread 0x0304c1 thru 0x03ffff (0xfb3f bytes) 000352 1777047345.155463 trunc from 0x040000 to 0x02ed66 (0x11299 bytes) 000353 1777047345.229159 write 0x022b9c thru 0x02cd8f (0xa1f4 bytes) 000354 1777047345.238003 read 0x029a7c thru 0x02ed66 (0x52eb bytes) 000355 1777047345.247886 trunc from 0x02ed67 to 0x005635 (0x29731 bytes) 000356 1777047345.333009 mapwrite 0x006298 thru 0x00b9fd (0x5766 bytes) 000357 1777047345.666813 mapread 0x006961 thru 0x00b9fd (0x509d bytes) 000358 1777047345.674518 read 0x007412 thru 0x00b9fd (0x45ec bytes) 000359 1777047345.676189 mapread 0x003af8 thru 0x00449e (0x9a7 bytes) 000360 1777047345.714867 MIRROR_RESYNC 12 mirrors 000362 1777047353.950376 trunc from 0x00b9fe to 0x02a02c (0x1e62f bytes) 000363 1777047354.380475 read 0x0187ac thru 0x022e43 (0xa698 bytes) 000366 1777047354.430786 write 0x018d4a thru 0x020fa1 (0x8258 bytes) 000367 1777047354.445112 read 0x00149b thru 0x006f36 (0x5a9c bytes) 000371 1777047354.516058 mapwrite 0x035225 thru 0x0366bd (0x1499 bytes) 000372 1777047355.004422 mapread 0x01cfea thru 0x02c6e9 (0xf700 bytes) 000373 1777047355.033288 write 0x02efa6 thru 0x034a20 (0x5a7b bytes) 000374 1777047355.077457 mapwrite 0x036612 thru 0x03bf8b (0x597a bytes) 000375 1777047355.635143 trunc from 0x03bf8c to 0x03d40d (0x1482 bytes) 000377 1777047355.722828 trunc from 0x03d40e to 0x018bfd (0x24810 bytes) 000378 1777047355.867041 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x18bfe): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18bfe): Mirror 2: 0x388f5298 (stale) Mirror 3: 0x388f5298 (stale) Mirror 5: 0x388f5298 (stale) Mirror 6: 0x388f5298 (stale) Mirror 7: 0x388f5298 (stale) Mirror 8: 0x388f5298 (stale) Mirror 9: 0x388f5298 (stale) Mirror 10: 0x388f5298 (stale) Mirror 11: 0x388f5298 (stale) Mirror 12: 0x388f5298 (stale) Mirror 13: 0x388f5298 (stale) CRC-32 checksum value for chunk [0, 0x18bfe): Mirror 1: 0x388f5298 Mirror 14: 0x388f5298 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18bfe: skipped 000380 1777047360.187720 mapwrite 0x030a3f thru 0x03e32d (0xd8ef bytes) 000383 1777047361.019598 read 0x00b8de thru 0x0170fb (0xb81e bytes) 000384 1777047361.071255 write 0x02aa5d thru 0x02e77d (0x3d21 bytes) 000386 1777047361.167964 trunc from 0x03e32e to 0x039c3f (0x46ee bytes) 000387 1777047361.296406 trunc from 0x039c40 to 0x033c9c (0x5fa3 bytes) 000388 1777047361.417781 write 0x002b12 thru 0x00583b (0x2d2a bytes) 000389 1777047361.519535 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x33c9d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33c9d): Mirror 2: 0xd25f67 (stale) Mirror 3: 0xd25f67 (stale) Mirror 5: 0xd25f67 (stale) Mirror 6: 0xd25f67 (stale) Mirror 7: 0xd25f67 (stale) Mirror 8: 0xd25f67 (stale) Mirror 9: 0xd25f67 (stale) Mirror 10: 0xd25f67 (stale) Mirror 11: 0xd25f67 (stale) Mirror 12: 0xd25f67 (stale) Mirror 13: 0xd25f67 (stale) Mirror 14: 0xd25f67 (stale) CRC-32 checksum value for chunk [0, 0x33c9d): Mirror 1: 0x909614e4 Mirror 15: 0x909614e4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33c9d: skipped 000391 1777047366.537151 read 0x0222a1 thru 0x0301e3 (0xdf43 bytes) 000393 1777047366.681253 mapwrite 0x00bbe3 thru 0x0167ef (0xac0d bytes) 000396 1777047367.646492 mapwrite 0x028c88 thru 0x02d31d (0x4696 bytes) 000397 1777047367.952207 read 0x03212d thru 0x032203 (0xd7 bytes) 000398 1777047368.039214 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33c9d): Mirror 3: 0xd25f67 (stale) Mirror 5: 0xd25f67 (stale) Mirror 6: 0xd25f67 (stale) Mirror 7: 0xd25f67 (stale) Mirror 8: 0xd25f67 (stale) Mirror 9: 0xd25f67 (stale) Mirror 10: 0xd25f67 (stale) Mirror 11: 0xd25f67 (stale) Mirror 12: 0xd25f67 (stale) Mirror 13: 0xd25f67 (stale) Mirror 14: 0xd25f67 (stale) Mirror 15: 0x909614e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33c9d): Mirror 3: 0xd25f67 (stale) Mirror 5: 0xd25f67 (stale) Mirror 6: 0xd25f67 (stale) Mirror 7: 0xd25f67 (stale) Mirror 8: 0xd25f67 (stale) Mirror 9: 0xd25f67 (stale) Mirror 10: 0xd25f67 (stale) Mirror 11: 0xd25f67 (stale) Mirror 12: 0xd25f67 (stale) Mirror 13: 0xd25f67 (stale) Mirror 14: 0xd25f67 (stale) Mirror 15: 0x909614e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33c9d: skipped 000399 1777047370.608670 trunc from 0x033c9d to 0x00e0e2 (0x25bba bytes) 000400 1777047370.679180 trunc from 0x00e0e3 to 0x021994 (0x138b2 bytes) 000403 1777047370.762076 trunc from 0x021995 to 0x017db9 (0x9bdb bytes) 000404 1777047370.837223 trunc from 0x017dba to 0x017fec (0x233 bytes) 000406 1777047370.965162 write 0x01b9c7 thru 0x02414f (0x8789 bytes) 000407 1777047370.981533 read 0x01aae0 thru 0x02414f (0x9670 bytes) 000408 1777047371.006612 mapwrite 0x00355d thru 0x007c37 (0x46db bytes) 000409 1777047371.643548 MIRROR_SPLIT mirror 3 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24150): Mirror 3: 0x8c85f3a1 (stale) Mirror 5: 0x8c85f3a1 (stale) Mirror 7: 0x8c85f3a1 (stale) Mirror 8: 0x8c85f3a1 (stale) Mirror 9: 0x8c85f3a1 (stale) Mirror 10: 0x8c85f3a1 (stale) Mirror 11: 0x8c85f3a1 (stale) Mirror 12: 0x8c85f3a1 (stale) Mirror 13: 0x8c85f3a1 (stale) Mirror 14: 0x8c85f3a1 (stale) Mirror 15: 0x16c9fe0f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24150): Mirror 3: 0x8c85f3a1 (stale) Mirror 5: 0x8c85f3a1 (stale) Mirror 7: 0x8c85f3a1 (stale) Mirror 8: 0x8c85f3a1 (stale) Mirror 9: 0x8c85f3a1 (stale) Mirror 10: 0x8c85f3a1 (stale) Mirror 11: 0x8c85f3a1 (stale) Mirror 12: 0x8c85f3a1 (stale) Mirror 13: 0x8c85f3a1 (stale) Mirror 14: 0x8c85f3a1 (stale) Mirror 15: 0x16c9fe0f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24150: skipped 000410 1777047374.936629 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24150): Mirror 3: 0x8c85f3a1 (stale) Mirror 5: 0x8c85f3a1 (stale) Mirror 7: 0x8c85f3a1 (stale) Mirror 8: 0x8c85f3a1 (stale) Mirror 9: 0x8c85f3a1 (stale) Mirror 10: 0x8c85f3a1 (stale) Mirror 11: 0x8c85f3a1 (stale) Mirror 12: 0x8c85f3a1 (stale) Mirror 13: 0x8c85f3a1 (stale) Mirror 14: 0x8c85f3a1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24150): Mirror 3: 0x8c85f3a1 (stale) Mirror 5: 0x8c85f3a1 (stale) Mirror 7: 0x8c85f3a1 (stale) Mirror 8: 0x8c85f3a1 (stale) Mirror 9: 0x8c85f3a1 (stale) Mirror 10: 0x8c85f3a1 (stale) Mirror 11: 0x8c85f3a1 (stale) Mirror 12: 0x8c85f3a1 (stale) Mirror 13: 0x8c85f3a1 (stale) Mirror 14: 0x8c85f3a1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24150: skipped 000411 1777047377.866000 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x24150): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24150): Mirror 3: 0x8c85f3a1 (stale) Mirror 5: 0x8c85f3a1 (stale) Mirror 7: 0x8c85f3a1 (stale) Mirror 8: 0x8c85f3a1 (stale) Mirror 9: 0x8c85f3a1 (stale) Mirror 10: 0x8c85f3a1 (stale) Mirror 11: 0x8c85f3a1 (stale) Mirror 12: 0x8c85f3a1 (stale) Mirror 13: 0x8c85f3a1 (stale) Mirror 14: 0x8c85f3a1 (stale) CRC-32 checksum value for chunk [0, 0x24150): Mirror 1: 0x41bb2b54 Mirror 15: 0x41bb2b54 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24150: skipped 000415 1777047382.008464 mapwrite 0x01c30e thru 0x01d649 (0x133c bytes) 000416 1777047382.869784 MIRROR_RESYNC 12 mirrors 000417 1777047390.372781 mapwrite 0x01d498 thru 0x024514 (0x707d bytes) 000419 1777047391.218080 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x24515): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24515): Mirror 3: 0x36ff8c3c (stale) Mirror 5: 0x36ff8c3c (stale) Mirror 7: 0x36ff8c3c (stale) Mirror 8: 0x36ff8c3c (stale) Mirror 9: 0x36ff8c3c (stale) Mirror 10: 0x36ff8c3c (stale) Mirror 11: 0x36ff8c3c (stale) Mirror 12: 0x36ff8c3c (stale) Mirror 13: 0x36ff8c3c (stale) Mirror 14: 0x36ff8c3c (stale) Mirror 15: 0x36ff8c3c (stale) CRC-32 checksum value for chunk [0, 0x24515): Mirror 1: 0x69eadd1e Mirror 16: 0x69eadd1e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24515: skipped 000420 1777047396.172759 mapread 0x015d2b thru 0x015eaf (0x185 bytes) 000422 1777047396.345724 mapwrite 0x01626e thru 0x01e76b (0x84fe bytes) 000423 1777047397.374098 mapwrite 0x011e8e thru 0x021b57 (0xfcca bytes) 000425 1777047397.758844 read 0x000a2b thru 0x0051eb (0x47c1 bytes) 000426 1777047397.830429 write 0x00e89e thru 0x01321b (0x497e bytes) 000427 1777047397.921699 read 0x001948 thru 0x01045b (0xeb14 bytes) 000431 1777047397.993531 mapwrite 0x00cf4b thru 0x01072b (0x37e1 bytes) 000433 1777047398.357218 mapwrite 0x03ccf0 thru 0x03ffff (0x3310 bytes) 000434 1777047398.720294 mapread 0x021785 thru 0x02fd74 (0xe5f0 bytes) 000438 1777047398.751814 write 0x03b7b5 thru 0x03ffff (0x484b bytes) 000439 1777047398.756456 mapread 0x038d80 thru 0x0394c6 (0x747 bytes) 000441 1777047398.770386 read 0x01f410 thru 0x02f257 (0xfe48 bytes) 000444 1777047398.783502 write 0x000b81 thru 0x00415b (0x35db bytes) 000445 1777047398.792733 mapread 0x012bfa thru 0x01b927 (0x8d2e bytes) 000446 1777047398.800043 mapread 0x029635 thru 0x0366c0 (0xd08c bytes) 000447 1777047398.811308 mapread 0x03e99a thru 0x03ffff (0x1666 bytes) 000450 1777047398.852526 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x28e6fa58 (stale) Mirror 5: 0x28e6fa58 (stale) Mirror 7: 0x28e6fa58 (stale) Mirror 8: 0x28e6fa58 (stale) Mirror 9: 0x28e6fa58 (stale) Mirror 10: 0x28e6fa58 (stale) Mirror 11: 0x28e6fa58 (stale) Mirror 12: 0x28e6fa58 (stale) Mirror 13: 0x28e6fa58 (stale) Mirror 14: 0x28e6fa58 (stale) Mirror 15: 0x28e6fa58 (stale) Mirror 16: 0xb2d249e0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf9c0fca0 Mirror 17: 0xf9c0fca0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000451 1777047402.991135 MIRROR_RESYNC 14 mirrors 000455 1777047410.539027 read 0x013d0d thru 0x01e19a (0xa48e bytes) 000457 1777047410.604857 MIRROR_EXTEND to 15 mirrors 000458 1777047413.562730 MIRROR_EXTEND to 16 mirrors truncating to largest ever: 0x3ff39 000460 1777047416.757145 trunc from 0x040000 to 0x03ff38 (0xc7 bytes) 000461 1777047417.429965 write 0x03f5ab thru 0x03ffff (0xa55 bytes) 000464 1777047417.486644 trunc from 0x040000 to 0x03b820 (0x47df bytes) 000466 1777047417.693279 mapwrite 0x03a8eb thru 0x03ffff (0x5715 bytes) 000468 1777047418.063789 write 0x03d9d8 thru 0x03ffff (0x2628 bytes) 000469 1777047418.069998 mapwrite 0x0334b7 thru 0x036d3d (0x3887 bytes) 000470 1777047418.386305 mapread 0x03caaf thru 0x03eed2 (0x2424 bytes) 000474 1777047418.435923 trunc from 0x040000 to 0x024693 (0x1b96c bytes) 000476 1777047418.573430 trunc from 0x024694 to 0x0221b6 (0x24dd bytes) 000479 1777047418.665593 mapwrite 0x03d8b1 thru 0x03ffff (0x274f bytes) 000480 1777047419.064965 mapread 0x009f85 thru 0x00eace (0x4b4a bytes) 000482 1777047419.184704 mapread 0x01beef thru 0x028081 (0xc193 bytes) 000485 1777047419.356101 mapwrite 0x011dfa thru 0x016362 (0x4569 bytes) 000487 1777047419.742700 write 0x03fd33 thru 0x03ffff (0x2cd bytes) 000488 1777047419.762745 read 0x026851 thru 0x035ead (0xf65d bytes) 000490 1777047419.851611 mapwrite 0x005222 thru 0x00579a (0x579 bytes) 000494 1777047420.110732 trunc from 0x040000 to 0x015d9e (0x2a261 bytes) 000495 1777047420.175489 mapwrite 0x003d28 thru 0x00b12c (0x7405 bytes) 000497 1777047420.427488 read 0x0064b5 thru 0x00a976 (0x44c2 bytes) 000498 1777047420.441577 trunc from 0x015d9f to 0x007f90 (0xde0e bytes) 000499 1777047420.621472 trunc from 0x007f91 to 0x024b53 (0x1cbc3 bytes) 000501 1777047420.737659 trunc from 0x024b54 to 0x02fecc (0xb379 bytes) 000505 1777047420.866894 write 0x02bd52 thru 0x03bb77 (0xfe26 bytes) 000506 1777047421.013950 read 0x01ffc5 thru 0x02a447 (0xa483 bytes) 000509 1777047421.063843 mapread 0x038722 thru 0x03bb77 (0x3456 bytes) 000510 1777047421.084899 trunc from 0x03bb78 to 0x0301a1 (0xb9d6 bytes) 000511 1777047421.204836 read 0x00a6b2 thru 0x00c3db (0x1d2a bytes) 000513 1777047421.263792 MIRROR_RESYNC 16 mirrors 000515 1777047429.590246 mapread 0x02646c thru 0x0301a1 (0x9d36 bytes) 000516 1777047429.715276 MIRROR_SPLIT mirror 6 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xba:0x0]: Invalid argument (22) 000517 1777047432.182474 write 0x03b080 thru 0x03ffff (0x4f80 bytes) 000518 1777047432.526209 trunc from 0x040000 to 0x01ea2b (0x215d4 bytes) 000519 1777047432.615363 write 0x00e213 thru 0x0103b6 (0x21a4 bytes) 000520 1777047432.647989 read 0x00ace2 thru 0x00fd6f (0x508e bytes) 000522 1777047432.691304 read 0x0020bd thru 0x011eb5 (0xfdf9 bytes) 000525 1777047432.836219 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ea2c): Mirror 5: 0xabd771a5 (stale) Mirror 7: 0xabd771a5 (stale) Mirror 8: 0xabd771a5 (stale) Mirror 9: 0xabd771a5 (stale) Mirror 11: 0xabd771a5 (stale) Mirror 12: 0xabd771a5 (stale) Mirror 13: 0xabd771a5 (stale) Mirror 14: 0xabd771a5 (stale) Mirror 15: 0xabd771a5 (stale) Mirror 16: 0xabd771a5 (stale) Mirror 17: 0xabd771a5 (stale) Mirror 18: 0xabd771a5 (stale) Mirror 19: 0xabd771a5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ea2c): Mirror 5: 0xabd771a5 (stale) Mirror 7: 0xabd771a5 (stale) Mirror 8: 0xabd771a5 (stale) Mirror 9: 0xabd771a5 (stale) Mirror 11: 0xabd771a5 (stale) Mirror 12: 0xabd771a5 (stale) Mirror 13: 0xabd771a5 (stale) Mirror 14: 0xabd771a5 (stale) Mirror 15: 0xabd771a5 (stale) Mirror 16: 0xabd771a5 (stale) Mirror 17: 0xabd771a5 (stale) Mirror 18: 0xabd771a5 (stale) Mirror 19: 0xabd771a5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ea2c: skipped 000526 1777047436.235256 MIRROR_SPLIT mirror 12 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ea2c): Mirror 5: 0xabd771a5 (stale) Mirror 7: 0xabd771a5 (stale) Mirror 8: 0xabd771a5 (stale) Mirror 9: 0xabd771a5 (stale) Mirror 11: 0xabd771a5 (stale) Mirror 12: 0xabd771a5 (stale) Mirror 13: 0xabd771a5 (stale) Mirror 14: 0xabd771a5 (stale) Mirror 15: 0xabd771a5 (stale) Mirror 16: 0xabd771a5 (stale) Mirror 17: 0xabd771a5 (stale) Mirror 19: 0xabd771a5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ea2c): Mirror 5: 0xabd771a5 (stale) Mirror 7: 0xabd771a5 (stale) Mirror 8: 0xabd771a5 (stale) Mirror 9: 0xabd771a5 (stale) Mirror 11: 0xabd771a5 (stale) Mirror 12: 0xabd771a5 (stale) Mirror 13: 0xabd771a5 (stale) Mirror 14: 0xabd771a5 (stale) Mirror 15: 0xabd771a5 (stale) Mirror 16: 0xabd771a5 (stale) Mirror 17: 0xabd771a5 (stale) Mirror 19: 0xabd771a5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ea2c: skipped 000527 1777047439.698089 trunc from 0x01ea2c to 0x01f6b4 (0xc89 bytes) 000528 1777047439.792272 mapread 0x001a6d thru 0x00e332 (0xc8c6 bytes) 000529 1777047439.985644 trunc from 0x01f6b5 to 0x01c4a7 (0x320d bytes) 000530 1777047440.068163 mapread 0x0092e3 thru 0x0152aa (0xbfc8 bytes) 000531 1777047440.134982 write 0x0352bd thru 0x03d90c (0x8650 bytes) 000533 1777047440.148857 mapread 0x0389e9 thru 0x03d90c (0x4f24 bytes) 000537 1777047440.186767 mapwrite 0x00e95b thru 0x015143 (0x67e9 bytes) 000538 1777047440.369456 trunc from 0x03d90d to 0x008a57 (0x34eb5 bytes) 000539 1777047440.419624 trunc from 0x008a58 to 0x02cd87 (0x24330 bytes) 000541 1777047440.509105 read 0x002be8 thru 0x009689 (0x6aa2 bytes) 000543 1777047440.558863 write 0x00d2bf thru 0x01b717 (0xe459 bytes) 000544 1777047440.646363 MIRROR_RESYNC 13 mirrors 000545 1777047447.004336 trunc from 0x02cd88 to 0x0193c9 (0x139be bytes) 000547 1777047447.425115 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x193ca): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x193ca): Mirror 5: 0x8cfcf4f8 (stale) Mirror 7: 0x8cfcf4f8 (stale) Mirror 8: 0x8cfcf4f8 (stale) Mirror 9: 0x8cfcf4f8 (stale) Mirror 11: 0x8cfcf4f8 (stale) Mirror 12: 0x8cfcf4f8 (stale) Mirror 13: 0x8cfcf4f8 (stale) Mirror 14: 0x8cfcf4f8 (stale) Mirror 15: 0x8cfcf4f8 (stale) Mirror 16: 0x8cfcf4f8 (stale) Mirror 17: 0x8cfcf4f8 (stale) Mirror 19: 0x8cfcf4f8 (stale) CRC-32 checksum value for chunk [0, 0x193ca): Mirror 1: 0x8cfcf4f8 Mirror 20: 0x8cfcf4f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x193ca: skipped 000548 1777047451.846081 trunc from 0x0193ca to 0x008977 (0x10a52 bytes) 000551 1777047452.345333 mapwrite 0x035c16 thru 0x03d826 (0x7c11 bytes) 000554 1777047452.864631 read 0x022850 thru 0x02b06d (0x881e bytes) 000557 1777047452.962976 mapread 0x0328d3 thru 0x03d826 (0xaf54 bytes) 000558 1777047452.986316 trunc from 0x03d827 to 0x035f69 (0x78bd bytes) 000559 1777047453.086730 write 0x03cfdc thru 0x03ffff (0x3024 bytes) 000560 1777047453.090850 mapread 0x01a5be thru 0x028738 (0xe17b bytes) 000561 1777047453.139257 mapread 0x01c7e9 thru 0x01db07 (0x131f bytes) 000562 1777047453.183026 MIRROR_RESYNC 14 mirrors 000563 1777047460.630459 MIRROR_EXTEND to 15 mirrors 000566 1777047464.310511 mapread 0x016fc7 thru 0x02556d (0xe5a7 bytes) 000567 1777047464.488234 mapread 0x0090e5 thru 0x0134e7 (0xa403 bytes) 000570 1777047464.548483 mapread 0x02f619 thru 0x039365 (0x9d4d bytes) 000571 1777047464.656650 mapread 0x03ab27 thru 0x03e7e6 (0x3cc0 bytes) 000573 1777047464.661551 trunc from 0x040000 to 0x03ed65 (0x129a bytes) 000574 1777047465.256441 MIRROR_RESYNC 15 mirrors 000575 1777047472.403933 mapread 0x0352a3 thru 0x03ed65 (0x9ac3 bytes) 000577 1777047472.473231 write 0x029beb thru 0x02a6a2 (0xab8 bytes) 000578 1777047472.869043 read 0x038797 thru 0x039a3c (0x12a6 bytes) 000579 1777047472.930184 mapread 0x00fb6b thru 0x01b9c7 (0xbe5d bytes) 000582 1777047473.085976 write 0x0203a3 thru 0x02628d (0x5eeb bytes) 000583 1777047473.133257 MIRROR_RESYNC 15 mirrors 000584 1777047479.923414 trunc from 0x03ed66 to 0x0264fd (0x18868 bytes) 000585 1777047480.242274 read 0x006bdc thru 0x00f7f9 (0x8c1e bytes) 000586 1777047480.304888 write 0x03c455 thru 0x03ffff (0x3bab bytes) 000587 1777047480.313406 mapread 0x013ed2 thru 0x014b83 (0xcb2 bytes) 000590 1777047480.403757 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x438cc834 (stale) Mirror 7: 0x438cc834 (stale) Mirror 8: 0x438cc834 (stale) Mirror 9: 0x438cc834 (stale) Mirror 11: 0x438cc834 (stale) Mirror 12: 0x438cc834 (stale) Mirror 13: 0x438cc834 (stale) Mirror 14: 0x438cc834 (stale) Mirror 15: 0x438cc834 (stale) Mirror 16: 0x438cc834 (stale) Mirror 17: 0x438cc834 (stale) Mirror 19: 0x438cc834 (stale) Mirror 20: 0x438cc834 (stale) Mirror 21: 0x438cc834 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6ec9fd9e Mirror 22: 0x6ec9fd9e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000591 1777047484.248795 read 0x007861 thru 0x0155d2 (0xdd72 bytes) 000592 1777047484.309810 MIRROR_SPLIT mirror 6 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x438cc834 (stale) Mirror 7: 0x438cc834 (stale) Mirror 8: 0x438cc834 (stale) Mirror 9: 0x438cc834 (stale) Mirror 11: 0x438cc834 (stale) Mirror 13: 0x438cc834 (stale) Mirror 14: 0x438cc834 (stale) Mirror 15: 0x438cc834 (stale) Mirror 16: 0x438cc834 (stale) Mirror 17: 0x438cc834 (stale) Mirror 19: 0x438cc834 (stale) Mirror 20: 0x438cc834 (stale) Mirror 21: 0x438cc834 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x438cc834 (stale) Mirror 7: 0x438cc834 (stale) Mirror 8: 0x438cc834 (stale) Mirror 9: 0x438cc834 (stale) Mirror 11: 0x438cc834 (stale) Mirror 13: 0x438cc834 (stale) Mirror 14: 0x438cc834 (stale) Mirror 15: 0x438cc834 (stale) Mirror 16: 0x438cc834 (stale) Mirror 17: 0x438cc834 (stale) Mirror 19: 0x438cc834 (stale) Mirror 20: 0x438cc834 (stale) Mirror 21: 0x438cc834 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6ec9fd9e Mirror 22: 0x6ec9fd9e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000594 1777047489.615278 write 0x0224b8 thru 0x0296be (0x7207 bytes) 000595 1777047490.142655 mapwrite 0x00af76 thru 0x00c81d (0x18a8 bytes) 000596 1777047490.372911 write 0x003cea thru 0x00aa0c (0x6d23 bytes) 000598 1777047490.380825 mapwrite 0x007ec9 thru 0x00f6d5 (0x780d bytes) 000600 1777047490.872743 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x438cc834 (stale) Mirror 7: 0x438cc834 (stale) Mirror 8: 0x438cc834 (stale) Mirror 9: 0x438cc834 (stale) Mirror 11: 0x438cc834 (stale) Mirror 13: 0x438cc834 (stale) Mirror 14: 0x438cc834 (stale) Mirror 15: 0x438cc834 (stale) Mirror 16: 0x438cc834 (stale) Mirror 17: 0x438cc834 (stale) Mirror 19: 0x438cc834 (stale) Mirror 20: 0x438cc834 (stale) Mirror 21: 0x438cc834 (stale) Mirror 22: 0x6ec9fd9e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xed41da2c Mirror 23: 0xed41da2c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000603 1777047494.458548 mapread 0x03f7c6 thru 0x03faca (0x305 bytes) 000606 1777047494.628623 read 0x0303c2 thru 0x03546e (0x50ad bytes) 000612 1777047494.928562 MIRROR_RESYNC 16 mirrors 000614 1777047502.172139 write 0x0297a5 thru 0x039646 (0xfea2 bytes) 000617 1777047502.589848 write 0x02d9f1 thru 0x034ead (0x74bd bytes) 000619 1777047502.644706 MIRROR_RESYNC 16 mirrors 000621 1777047511.645466 mapwrite 0x00c7be thru 0x0154e6 (0x8d29 bytes) 000623 1777047512.424492 MIRROR_SPLIT mirror 5 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x1b3ff7d5 (stale) Mirror 7: 0x1b3ff7d5 (stale) Mirror 8: 0x1b3ff7d5 (stale) Mirror 9: 0x1b3ff7d5 (stale) Mirror 13: 0x1b3ff7d5 (stale) Mirror 14: 0x1b3ff7d5 (stale) Mirror 15: 0x1b3ff7d5 (stale) Mirror 16: 0x1b3ff7d5 (stale) Mirror 17: 0x1b3ff7d5 (stale) Mirror 19: 0x1b3ff7d5 (stale) Mirror 20: 0x1b3ff7d5 (stale) Mirror 21: 0x1b3ff7d5 (stale) Mirror 22: 0x1b3ff7d5 (stale) Mirror 23: 0x1b3ff7d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x1b3ff7d5 (stale) Mirror 7: 0x1b3ff7d5 (stale) Mirror 8: 0x1b3ff7d5 (stale) Mirror 9: 0x1b3ff7d5 (stale) Mirror 13: 0x1b3ff7d5 (stale) Mirror 14: 0x1b3ff7d5 (stale) Mirror 15: 0x1b3ff7d5 (stale) Mirror 16: 0x1b3ff7d5 (stale) Mirror 17: 0x1b3ff7d5 (stale) Mirror 19: 0x1b3ff7d5 (stale) Mirror 20: 0x1b3ff7d5 (stale) Mirror 21: 0x1b3ff7d5 (stale) Mirror 22: 0x1b3ff7d5 (stale) Mirror 23: 0x1b3ff7d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000624 1777047514.826430 mapwrite 0x0097e1 thru 0x01428a (0xaaaa bytes) 000626 1777047515.047571 read 0x03496b thru 0x03d2b7 (0x894d bytes) 000627 1777047515.105878 MIRROR_SPLIT mirror 7 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x1b3ff7d5 (stale) Mirror 7: 0x1b3ff7d5 (stale) Mirror 8: 0x1b3ff7d5 (stale) Mirror 9: 0x1b3ff7d5 (stale) Mirror 13: 0x1b3ff7d5 (stale) Mirror 14: 0x1b3ff7d5 (stale) Mirror 16: 0x1b3ff7d5 (stale) Mirror 17: 0x1b3ff7d5 (stale) Mirror 19: 0x1b3ff7d5 (stale) Mirror 20: 0x1b3ff7d5 (stale) Mirror 21: 0x1b3ff7d5 (stale) Mirror 22: 0x1b3ff7d5 (stale) Mirror 23: 0x1b3ff7d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x1b3ff7d5 (stale) Mirror 7: 0x1b3ff7d5 (stale) Mirror 8: 0x1b3ff7d5 (stale) Mirror 9: 0x1b3ff7d5 (stale) Mirror 13: 0x1b3ff7d5 (stale) Mirror 14: 0x1b3ff7d5 (stale) Mirror 16: 0x1b3ff7d5 (stale) Mirror 17: 0x1b3ff7d5 (stale) Mirror 19: 0x1b3ff7d5 (stale) Mirror 20: 0x1b3ff7d5 (stale) Mirror 21: 0x1b3ff7d5 (stale) Mirror 22: 0x1b3ff7d5 (stale) Mirror 23: 0x1b3ff7d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000629 1777047517.300439 trunc from 0x040000 to 0x01663d (0x299c2 bytes) 000632 1777047517.450631 mapwrite 0x00b2f7 thru 0x0199bb (0xe6c5 bytes) 000633 1777047517.853434 trunc from 0x0199bc to 0x01209e (0x791d bytes) 000635 1777047517.987630 trunc from 0x01209f to 0x0128d2 (0x834 bytes) 000636 1777047518.115812 read 0x00bab1 thru 0x0128d2 (0x6e22 bytes) 000637 1777047518.122726 mapread 0x005a07 thru 0x0103b8 (0xa9b2 bytes) 000641 1777047518.199461 write 0x021675 thru 0x023ea6 (0x2832 bytes) 000642 1777047518.210032 write 0x00ba77 thru 0x017a55 (0xbfdf bytes) 000643 1777047518.240074 trunc from 0x023ea7 to 0x00f0cf (0x14dd7 bytes) 000644 1777047518.303203 read 0x007ae5 thru 0x00a151 (0x266d bytes) 000647 1777047518.313179 mapread 0x009973 thru 0x00f0cf (0x575d bytes) 000648 1777047518.318861 read 0x009bc3 thru 0x00f0cf (0x550d bytes) 000649 1777047518.326272 read 0x0099f7 thru 0x00f0cf (0x56d9 bytes) 000650 1777047518.334097 write 0x0304ee thru 0x03b7fc (0xb30f bytes) 000651 1777047518.338483 mapwrite 0x035d35 thru 0x03ffff (0xa2cb bytes) 000652 1777047518.728692 trunc from 0x040000 to 0x020a8c (0x1f573 bytes) 000653 1777047518.794869 mapread 0x007519 thru 0x00a5fa (0x30e2 bytes) 000654 1777047518.810478 write 0x001560 thru 0x005346 (0x3de7 bytes) 000655 1777047518.813707 write 0x0044fc thru 0x00a67e (0x6183 bytes) 000657 1777047518.833634 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x20a8d): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20a8d): Mirror 5: 0x3aefe51b (stale) Mirror 7: 0x3aefe51b (stale) Mirror 8: 0x3aefe51b (stale) Mirror 9: 0x3aefe51b (stale) Mirror 13: 0x3aefe51b (stale) Mirror 14: 0x3aefe51b (stale) Mirror 16: 0x3aefe51b (stale) Mirror 17: 0x3aefe51b (stale) Mirror 19: 0x3aefe51b (stale) Mirror 20: 0x3aefe51b (stale) Mirror 21: 0x3aefe51b (stale) Mirror 22: 0x3aefe51b (stale) Mirror 23: 0x3aefe51b (stale) CRC-32 checksum value for chunk [0, 0x20a8d): Mirror 1: 0x14f86f28 Mirror 24: 0x14f86f28 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20a8d: skipped 000658 1777047522.438163 trunc from 0x020a8d to 0x03c324 (0x1b898 bytes) 000659 1777047522.796840 mapwrite 0x00f7cf thru 0x015111 (0x5943 bytes) 000661 1777047523.091017 read 0x025652 thru 0x026576 (0xf25 bytes) 000662 1777047523.122406 trunc from 0x03c325 to 0x005078 (0x372ac bytes) 000664 1777047523.226777 mapwrite 0x01250a thru 0x018c98 (0x678f bytes) 000665 1777047523.553305 mapread 0x0047ec thru 0x0121a6 (0xd9bb bytes) 000667 1777047523.616851 write 0x03a2a9 thru 0x03a85e (0x5b6 bytes) 000668 1777047523.643444 trunc from 0x03a85f to 0x02beae (0xe9b0 bytes) 000669 1777047523.720002 trunc from 0x02beaf to 0x00ae3f (0x2106f bytes) 000670 1777047523.875607 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xae40): Mirror 7: 0x1266ffd7 (stale) Mirror 8: 0x1266ffd7 (stale) Mirror 9: 0x1266ffd7 (stale) Mirror 13: 0x1266ffd7 (stale) Mirror 14: 0x1266ffd7 (stale) Mirror 16: 0x1266ffd7 (stale) Mirror 17: 0x1266ffd7 (stale) Mirror 19: 0x1266ffd7 (stale) Mirror 20: 0x1266ffd7 (stale) Mirror 21: 0x1266ffd7 (stale) Mirror 22: 0x1266ffd7 (stale) Mirror 23: 0x1266ffd7 (stale) Mirror 24: 0xe4206e00 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xae40): Mirror 7: 0x1266ffd7 (stale) Mirror 8: 0x1266ffd7 (stale) Mirror 9: 0x1266ffd7 (stale) Mirror 13: 0x1266ffd7 (stale) Mirror 14: 0x1266ffd7 (stale) Mirror 16: 0x1266ffd7 (stale) Mirror 17: 0x1266ffd7 (stale) Mirror 19: 0x1266ffd7 (stale) Mirror 20: 0x1266ffd7 (stale) Mirror 21: 0x1266ffd7 (stale) Mirror 22: 0x1266ffd7 (stale) Mirror 23: 0x1266ffd7 (stale) Mirror 24: 0xe4206e00 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xae40: skipped 000672 1777047526.828285 read 0x005b2f thru 0x00ae3f (0x5311 bytes) 000673 1777047526.910666 write 0x039586 thru 0x03b965 (0x23e0 bytes) 000674 1777047526.933019 read 0x03595d thru 0x03b965 (0x6009 bytes) 000675 1777047526.996074 trunc from 0x03b966 to 0x0386b3 (0x32b2 bytes) 000679 1777047527.055154 read 0x01ecc6 thru 0x0207d9 (0x1b14 bytes) 000680 1777047527.057071 read 0x00256d thru 0x00592a (0x33be bytes) 000681 1777047527.071212 MIRROR_RESYNC 14 mirrors 000682 1777047535.172966 trunc from 0x0386b4 to 0x0013a7 (0x3730c bytes) 000683 1777047535.391690 write 0x02dd9b thru 0x02e6c7 (0x92d bytes) 000684 1777047535.408802 trunc from 0x02e6c8 to 0x00841e (0x262a9 bytes) 000685 1777047535.502714 MIRROR_SPLIT mirror 2 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x841f): Mirror 7: 0xcde921c9 (stale) Mirror 9: 0xcde921c9 (stale) Mirror 13: 0xcde921c9 (stale) Mirror 14: 0xcde921c9 (stale) Mirror 16: 0xcde921c9 (stale) Mirror 17: 0xcde921c9 (stale) Mirror 19: 0xcde921c9 (stale) Mirror 20: 0xcde921c9 (stale) Mirror 21: 0xcde921c9 (stale) Mirror 22: 0xcde921c9 (stale) Mirror 23: 0xcde921c9 (stale) Mirror 24: 0xcde921c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x841f): Mirror 7: 0xcde921c9 (stale) Mirror 9: 0xcde921c9 (stale) Mirror 13: 0xcde921c9 (stale) Mirror 14: 0xcde921c9 (stale) Mirror 16: 0xcde921c9 (stale) Mirror 17: 0xcde921c9 (stale) Mirror 19: 0xcde921c9 (stale) Mirror 20: 0xcde921c9 (stale) Mirror 21: 0xcde921c9 (stale) Mirror 22: 0xcde921c9 (stale) Mirror 23: 0xcde921c9 (stale) Mirror 24: 0xcde921c9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x841f: skipped 000686 1777047537.864142 mapwrite 0x034833 thru 0x03f31f (0xaaed bytes) 000687 1777047538.377493 MIRROR_RESYNC 13 mirrors