-----============= acceptance-small: sanity-flr ============----- Sat Aug 15 09:50:56 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg603-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 09:51:12 (1786801872) === oleg603-client.virtnet: executing check_config_client /mnt/lustre oleg603-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg603-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff980fc61a0800.idle_timeout=debug osc.lustre-OST0001-osc-ffff980fc61a0800.idle_timeout=debug disable quota as required oleg603-server: oleg603-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 09:51:48 (1786801908) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8402] == sanity-flr test 0a: lfs mirror create with -N option == 09:51:50 (1786801910) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:52:07 (1786801927) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:52:10 (1786801930) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:52:13 (1786801933) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:52:36 (1786801956) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:52:39 (1786801959) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 09:52:43 (1786801963) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff980fc61a0800.stats=clear osc.lustre-OST0001-osc-ffff980fc61a0800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:52:51 (1786801971) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:53:04 (1786801984) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:53:14 (1786801994) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 09:53:18 (1786801998) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:53:27 (1786802007) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 09:53:36 (1786802016) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:53:43 (1786802023) PASS 3 (6s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:53:50 (1786802030) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:53:56 (1786802036) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:54:04 (1786802044) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:54:10 (1786802050) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.776097 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880787 s, 11.9 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:54:19 (1786802059) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.439073 s, 38.2 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 09:54:28 (1786802068) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:54:42 (1786802082) Stopping /mnt/lustre-ost1 (opts:) on oleg603-server oleg603-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff980fc61a0800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff980fc61a0800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg603-server: oleg603-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg603-server: oleg603-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg603-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg603-server oleg603-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff980fc61a0800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff980fc61a0800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg603-server: oleg603-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg603-server: oleg603-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg603-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (65s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:55:46 (1786802146) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.214262 s, 19.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg603-server oleg603-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff980fc61a0800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff980fc61a0800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg603-server: oleg603-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg603-server: oleg603-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg603-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (36s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:56:22 (1786802182) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg603-server oleg603-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff980fc61a0800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff980fc61a0800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg603-server: oleg603-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg603-server: oleg603-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg603-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg603-server oleg603-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff980fc61a0800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff980fc61a0800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg603-server: oleg603-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg603-server: oleg603-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg603-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (64s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:57:26 (1786802246) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.33536 s, 12.1 MB/s -rw-r--r-- 1 root root 52428800 Aug 15 09:57 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.905s user 0m0.001s sys 0m0.159s Stopping /mnt/lustre-ost1 (opts:) on oleg603-server oleg603-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff980fc61a0800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff980fc61a0800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.928s user 0m0.001s sys 0m0.177s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg603-server: oleg603-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg603-server: oleg603-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg603-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg603-server oleg603-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff980fc61a0800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff980fc61a0800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.210s user 0m0.002s sys 0m0.179s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg603-server: oleg603-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg603-server: oleg603-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg603-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg603-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff980fc61a0800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff980fc61a0800.ost_server_uuid in FULL state after 0 sec PASS 33b (85s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:58:51 (1786802331) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:58:54 (1786802334) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:58:57 (1786802337) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 09:59:00 (1786802340) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 09:59:07 (1786802347) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.484939 s, 25.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:59:19 (1786802359) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300618 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396355 s, 26.5 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:59:25 (1786802365) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:59:32 (1786802372) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0019863 s, 516 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000785543 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00172675 s, 593 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00140515 s, 729 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 09:59:42 (1786802382) create test files with size 15760014 .. b7d545b6206be6836372317eadba2419 - 4c396f6bc9144438200b7baec48397fc - 1d0964bae9e1a1e61f0a891e48b035a7 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg603-client.virtnet /mnt/lustre (opts:) Starting client: oleg603-client.virtnet: -o user_xattr,flock 192.168.206.103@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 10:00:07 (1786802407) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:00:21 (1786802421) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:00:27 (1786802427) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954968 s, 11.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xac:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xad:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234461 s, 44.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xae:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 10:00:38 (1786802438) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.151825 s, 13.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.302628 s, 13.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.209065 s, 10.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 10:00:54 (1786802454) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 10:00:58 (1786802458) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:01:02 (1786802462) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022825 s, 45.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 10:01:10 (1786802470) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.299638 s, 10.5 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:01:19 (1786802479) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:01:27 (1786802487) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.260715 s, 40.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg603-client.virtnet /mnt/lustre (opts:) Starting client: oleg603-client.virtnet: -o user_xattr,flock 192.168.206.103@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:01:37 (1786802497) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff980fc7316000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.238489 s, 44.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg603-client.virtnet /mnt/lustre (opts:) Starting client: oleg603-client.virtnet: -o user_xattr,flock 192.168.206.103@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff980fc61a4800.xattr_cache=1 PASS 44e (12s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:01:49 (1786802509) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (32s) == sanity-flr test 46: Verify setstripe --copy option ==== 10:02:21 (1786802541) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (33s) == sanity-flr test 47: Verify mirror obj alloc =========== 10:02:54 (1786802574) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 10:02:57 (1786802577) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.296217 s, 10.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.127358 s, 24.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x7556d95 (nosync,stale) Mirror 2: 0x7556d95 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 766cee3766b0fb6d883735f3fc32e636 - mirror 1 checksum: 2924873c0caee6e543a348f8808398ab - mirror 2 checksum: 766cee3766b0fb6d883735f3fc32e636 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-15T10:03:00-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-15T10:03:00-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 766cee3766b0fb6d883735f3fc32e636 - mirror 1 checksum: 2924873c0caee6e543a348f8808398ab - mirror 2 checksum: 2924873c0caee6e543a348f8808398ab - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-15T10:03:00-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-15T10:03:00-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } PASS 48 (9s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:03:08 (1786802588) Failing mds1 on oleg603-server Stopping /mnt/lustre-mds1 (opts:) on oleg603-server 10:03:14 (1786802594) shut down facet: mds1 facet_host: oleg603-server facet_failover_host: oleg603-server Failover mds1 to oleg603-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg603-server: oleg603-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Started lustre-MDT0000 10:03:36 (1786802616) targets are mounted 10:03:36 (1786802616) facet_failover done oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xd:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314985 s, 33.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xd lmm_fid: [0x200000405:0xd:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } PASS 50A (42s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:03:50 (1786802630) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 10:03:53 (1786802633) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:03:55 (1786802635) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00887231 s, 1.8 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 10:04:02 (1786802642) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:04:05 (1786802645) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:04:08 (1786802648) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:04:31 (1786802671) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:04:59 (1786802699) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x40:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:05:22 (1786802722) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:05:45 (1786802745) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 10:05:51 (1786802751) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 56001 mirror split pid 56002 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 56001 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 56002 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 10:06:59 (1786802819) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 2821 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000007 1786802822.232497 write 0x02bcb2 thru 0x0342aa (0x85f9 bytes) 000009 1786802822.239066 write 0x0237cc thru 0x031c79 (0xe4ae bytes) 000010 1786802822.269408 read 0x0340fa thru 0x0342aa (0x1b1 bytes) 000012 1786802822.271559 mapread 0x017816 thru 0x01998a (0x2175 bytes) truncating to largest ever: 0x95f7 000013 1786802822.295316 trunc from 0x0342ab to 0x0095f6 (0x2acb4 bytes) 000015 1786802822.327383 write 0x03c8d6 thru 0x03ffff (0x372a bytes) 000016 1786802822.328554 write 0x03ce3e thru 0x03f6f4 (0x28b7 bytes) 000017 1786802822.342407 MIRROR_EXTEND to 2 mirrors 000021 1786802823.068263 read 0x00f1f7 thru 0x018d84 (0x9b8e bytes) 000022 1786802823.083502 read 0x000c18 thru 0x0090fb (0x84e4 bytes) 000024 1786802823.106099 mapwrite 0x018c66 thru 0x027dd7 (0xf172 bytes) 000027 1786802823.352705 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x15996acc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x984fe1a4 Mirror 3: 0x984fe1a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000028 1786802824.053524 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x15996acc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x984fe1a4 Mirror 3: 0x984fe1a4 Mirror 4: 0x984fe1a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000029 1786802824.770327 write 0x00db9a thru 0x01204a (0x44b1 bytes) truncating to largest ever: 0xfe78 000031 1786802824.851765 trunc from 0x040000 to 0x00fe77 (0x30188 bytes) 000032 1786802824.874278 write 0x00d785 thru 0x016a3c (0x92b8 bytes) 000034 1786802824.879538 mapread 0x0127a9 thru 0x016a3c (0x4294 bytes) 000037 1786802824.890515 MIRROR_RESYNC 4 mirrors 000038 1786802825.691172 write 0x02fe9e thru 0x03cf7f (0xd0e2 bytes) 000039 1786802825.759200 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cf80): Mirror 2: 0xcae501aa (stale) Mirror 4: 0xcae501aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cf80): Mirror 2: 0xcae501aa (stale) Mirror 4: 0xcae501aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cf80: skipped 000040 1786802826.140000 mapwrite 0x025d20 thru 0x0331f4 (0xd4d5 bytes) truncating to largest ever: 0x2a165 000041 1786802826.228377 trunc from 0x03cf80 to 0x02a164 (0x12e1b bytes) truncating to largest ever: 0x2cbbb 000042 1786802826.249768 trunc from 0x02a165 to 0x02cbba (0x2a56 bytes) 000043 1786802826.269759 read 0x01492b thru 0x01688c (0x1f62 bytes) truncating to largest ever: 0x34011 000044 1786802826.275650 trunc from 0x02cbbb to 0x034010 (0x7456 bytes) 000046 1786802826.311478 MIRROR_RESYNC 3 mirrors 000047 1786802827.152921 mapread 0x02f5e0 thru 0x03171f (0x2140 bytes) 000049 1786802827.180020 read 0x023f45 thru 0x02f05f (0xb11b bytes) 000052 1786802827.195886 MIRROR_RESYNC 3 mirrors 000054 1786802827.649222 read 0x033a63 thru 0x034010 (0x5ae bytes) 000055 1786802827.649653 write 0x034dac thru 0x035824 (0xa79 bytes) 000057 1786802827.752316 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x35825): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35825): Mirror 2: 0x63b63662 (stale) Mirror 4: 0x63b63662 (stale) CRC-32 checksum value for chunk [0, 0x35825): Mirror 1: 0x46ca03e7 Mirror 5: 0x46ca03e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35825: skipped 000058 1786802828.717042 trunc from 0x035825 to 0x003894 (0x31f90 bytes) 000059 1786802828.855800 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3895): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3895): Mirror 2: 0x9872570f (stale) Mirror 4: 0x9872570f (stale) Mirror 5: 0x9872570f (stale) CRC-32 checksum value for chunk [0, 0x3895): Mirror 1: 0x9872570f Mirror 6: 0x9872570f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3895: skipped 000061 1786802829.873954 read 0x00268d thru 0x003894 (0x1208 bytes) truncating to largest ever: 0x3aaeb 000062 1786802829.888108 trunc from 0x003895 to 0x03aaea (0x37256 bytes) 000063 1786802829.984504 read 0x00c58a thru 0x00c931 (0x3a8 bytes) 000064 1786802830.003098 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3aaeb): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aaeb): Mirror 2: 0xc4cd1d96 (stale) Mirror 4: 0xc4cd1d96 (stale) Mirror 5: 0x42372b44 (stale) Mirror 6: 0xca387526 (stale) CRC-32 checksum value for chunk [0, 0x3aaeb): Mirror 1: 0xca387526 Mirror 7: 0xca387526 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aaeb: skipped 000067 1786802830.979688 mapread 0x035667 thru 0x03aaea (0x5484 bytes) 000068 1786802831.023687 write 0x01ee4c thru 0x027e9c (0x9051 bytes) truncating to largest ever: 0x3bcf5 000069 1786802831.097661 trunc from 0x03aaeb to 0x03bcf4 (0x120a bytes) 000070 1786802831.119381 mapwrite 0x034b94 thru 0x03cc52 (0x80bf bytes) 000071 1786802831.204945 trunc from 0x03cc53 to 0x00a7f9 (0x32459 bytes) 000072 1786802831.243871 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xa7fa): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa7fa): Mirror 2: 0xe818e58d (stale) Mirror 4: 0xe818e58d (stale) Mirror 5: 0xe818e58d (stale) Mirror 6: 0xe818e58d (stale) Mirror 7: 0xe818e58d (stale) CRC-32 checksum value for chunk [0, 0xa7fa): Mirror 1: 0xe818e58d Mirror 8: 0xe818e58d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa7fa: skipped 000073 1786802832.286905 MIRROR_RESYNC 7 mirrors 000075 1786802833.408297 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) 000077 1786802833.844059 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) 000078 1786802834.332833 write 0x039c48 thru 0x03ffff (0x63b8 bytes) 000080 1786802834.437631 mapwrite 0x0046dc thru 0x00e878 (0xa19d bytes) 000082 1786802834.597413 read 0x036f59 thru 0x03c11f (0x51c7 bytes) 000083 1786802834.605759 trunc from 0x040000 to 0x011f7d (0x2e082 bytes) 000085 1786802834.657900 write 0x03bfe7 thru 0x03c1a1 (0x1bb bytes) 000087 1786802834.664049 write 0x03b1b7 thru 0x03ffff (0x4e49 bytes) 000088 1786802834.666019 mapwrite 0x00a5bd thru 0x0152ca (0xad0e bytes) 000091 1786802834.764405 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000092 1786802835.177229 trunc from 0x040000 to 0x01a4aa (0x25b55 bytes) 000093 1786802835.205040 MIRROR_RESYNC 4 mirrors 000095 1786802836.134313 MIRROR_EXTEND to 5 mirrors 000096 1786802836.850198 write 0x03a1aa thru 0x03ffff (0x5e56 bytes) 000097 1786802836.980216 MIRROR_RESYNC 5 mirrors 000099 1786802838.113362 write 0x015bf9 thru 0x0225a5 (0xc9ad bytes) 000100 1786802838.223171 mapwrite 0x005008 thru 0x00e54a (0x9543 bytes) 000102 1786802838.338611 mapread 0x03dd13 thru 0x03ffff (0x22ed bytes) 000106 1786802838.357045 mapwrite 0x02e978 thru 0x0378f4 (0x8f7d bytes) 000108 1786802838.441851 mapwrite 0x011285 thru 0x01517f (0x3efb bytes) 000110 1786802838.518248 trunc from 0x040000 to 0x01d97e (0x22681 bytes) 000111 1786802838.563219 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1d97f): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d97f): Mirror 6: 0x34da576b (stale) Mirror 7: 0x34da576b (stale) Mirror 8: 0x34da576b (stale) Mirror 9: 0x34da576b (stale) CRC-32 checksum value for chunk [0, 0x1d97f): Mirror 1: 0x2e6878bb Mirror 10: 0x2e6878bb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d97f: skipped 000113 1786802839.404818 mapwrite 0x02e5ed thru 0x037c91 (0x96a5 bytes) 000114 1786802839.585963 write 0x0036e2 thru 0x006609 (0x2f28 bytes) 000116 1786802839.599045 read 0x0106bc thru 0x0145f7 (0x3f3c bytes) 000117 1786802839.609570 write 0x00d98c thru 0x0135f3 (0x5c68 bytes) 000118 1786802839.615244 mapread 0x013118 thru 0x019de5 (0x6cce bytes) 000119 1786802839.625153 trunc from 0x037c92 to 0x02ab0e (0xd183 bytes) 000120 1786802839.651675 mapwrite 0x03e573 thru 0x03ffff (0x1a8d bytes) 000121 1786802839.732765 mapread 0x0004ff thru 0x009c40 (0x9742 bytes) 000122 1786802839.750010 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd8d7d396 (stale) Mirror 8: 0xd8d7d396 (stale) Mirror 9: 0xd8d7d396 (stale) Mirror 10: 0xfe5333d0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd8d7d396 (stale) Mirror 8: 0xd8d7d396 (stale) Mirror 9: 0xd8d7d396 (stale) Mirror 10: 0xfe5333d0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000125 1786802840.186966 trunc from 0x040000 to 0x00a840 (0x357bf bytes) 000126 1786802840.204541 read 0x004cd0 thru 0x00a840 (0x5b71 bytes) 000128 1786802840.212748 write 0x00d97a thru 0x015f7d (0x8604 bytes) 000130 1786802840.216851 trunc from 0x015f7e to 0x03088d (0x1a910 bytes) 000131 1786802840.231640 write 0x0097f0 thru 0x00fbb5 (0x63c6 bytes) 000132 1786802840.233287 write 0x014556 thru 0x01ff29 (0xb9d4 bytes) 000133 1786802840.240283 trunc from 0x03088e to 0x006c8b (0x29c02 bytes) 000134 1786802840.254915 write 0x008e42 thru 0x0105b8 (0x7777 bytes) 000135 1786802840.256941 read 0x0019c5 thru 0x00b7f6 (0x9e32 bytes) 000136 1786802840.266493 trunc from 0x0105b9 to 0x02e140 (0x1db88 bytes) 000139 1786802840.291953 trunc from 0x02e141 to 0x00eb7a (0x1f5c6 bytes) 000140 1786802840.339850 read 0x001f0d thru 0x00eb7a (0xcc6e bytes) 000142 1786802840.341170 read 0x000d41 thru 0x00a887 (0x9b47 bytes) 000143 1786802840.346120 read 0x0061d3 thru 0x00eb7a (0x89a8 bytes) 000145 1786802840.347959 write 0x034074 thru 0x03bb2e (0x7abb bytes) 000146 1786802840.350642 mapwrite 0x032f78 thru 0x03ffff (0xd088 bytes) 000147 1786802840.514970 write 0x0201f2 thru 0x02c292 (0xc0a1 bytes) 000148 1786802840.521101 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd8d7d396 (stale) Mirror 8: 0xd8d7d396 (stale) Mirror 10: 0xfe5333d0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd8d7d396 (stale) Mirror 8: 0xd8d7d396 (stale) Mirror 10: 0xfe5333d0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000149 1786802840.951284 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd8d7d396 (stale) Mirror 8: 0xd8d7d396 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd8d7d396 (stale) Mirror 8: 0xd8d7d396 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000150 1786802841.374547 mapwrite 0x02befe thru 0x02d8cf (0x19d2 bytes) 000151 1786802841.442635 mapwrite 0x008e9b thru 0x00df70 (0x50d6 bytes) 000153 1786802841.530972 read 0x0261c4 thru 0x035c58 (0xfa95 bytes) 000154 1786802841.533051 write 0x0133f8 thru 0x021318 (0xdf21 bytes) 000156 1786802841.547685 write 0x00aeab thru 0x010b8e (0x5ce4 bytes) 000159 1786802841.551704 mapwrite 0x01455a thru 0x0232a9 (0xed50 bytes) 000161 1786802841.680405 trunc from 0x040000 to 0x0361b5 (0x9e4a bytes) 000163 1786802841.716652 MIRROR_RESYNC 3 mirrors 000164 1786802842.417986 read 0x023a7c thru 0x02f2f5 (0xb87a bytes) 000166 1786802842.430995 mapwrite 0x000093 thru 0x00adfe (0xad6c bytes) 000167 1786802842.756625 write 0x02010f thru 0x02c9c8 (0xc8ba bytes) 000168 1786802842.768658 read 0x0211a6 thru 0x025e83 (0x4cde bytes) 000169 1786802842.770058 mapread 0x005cc9 thru 0x00e949 (0x8c81 bytes) 000170 1786802842.772764 write 0x017fc2 thru 0x01b5e2 (0x3621 bytes) 000171 1786802842.787271 read 0x02d45c thru 0x02eb38 (0x16dd bytes) 000172 1786802842.788147 mapread 0x003000 thru 0x0095fe (0x65ff bytes) 000173 1786802842.790128 mapread 0x01c284 thru 0x020ea8 (0x4c25 bytes) 000174 1786802842.791944 write 0x0297f4 thru 0x030194 (0x69a1 bytes) 000176 1786802842.795176 write 0x02d9d5 thru 0x035791 (0x7dbd bytes) 000178 1786802842.800294 mapwrite 0x001359 thru 0x007a89 (0x6731 bytes) 000179 1786802842.864075 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x361b6): Mirror 7: 0x87ee7634 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x361b6): Mirror 7: 0x87ee7634 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x361b6: skipped 000180 1786802843.159248 trunc from 0x0361b6 to 0x015c0c (0x205a9 bytes) 000181 1786802843.190745 write 0x014e5d thru 0x01da75 (0x8c19 bytes) 000182 1786802843.209198 write 0x0231b1 thru 0x02e387 (0xb1d7 bytes) 000183 1786802843.220593 mapwrite 0x01652f thru 0x01c845 (0x6317 bytes) 000185 1786802843.338210 write 0x02d216 thru 0x02e5b1 (0x139c bytes) 000186 1786802843.339600 write 0x028b8c thru 0x0387a9 (0xfc1e bytes) 000187 1786802843.341852 mapwrite 0x01c394 thru 0x022077 (0x5ce4 bytes) 000189 1786802843.491253 mapread 0x02ff03 thru 0x0387a9 (0x88a7 bytes) 000192 1786802843.494385 write 0x01449d thru 0x015ad2 (0x1636 bytes) 000195 1786802843.499550 write 0x014603 thru 0x01d8b8 (0x92b6 bytes) 000198 1786802843.510216 mapwrite 0x03ff81 thru 0x03ffff (0x7f bytes) 000199 1786802843.713969 write 0x02f65a thru 0x03b878 (0xc21f bytes) 000201 1786802843.721943 write 0x01cf8c thru 0x027c7a (0xacef bytes) 000204 1786802843.734119 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x7662f8f1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4dcb449 Mirror 8: 0x4dcb449 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000205 1786802844.853597 mapwrite 0x03e3d6 thru 0x03ffff (0x1c2a bytes) 000210 1786802845.060133 mapread 0x0261d0 thru 0x02706d (0xe9e bytes) 000212 1786802845.078921 mapread 0x019ad2 thru 0x023e98 (0xa3c7 bytes) 000213 1786802845.094449 mapread 0x0163b1 thru 0x022495 (0xc0e5 bytes) 000215 1786802845.098552 mapwrite 0x00d6d7 thru 0x0103b4 (0x2cde bytes) 000216 1786802845.155622 mapwrite 0x01e5f6 thru 0x02d39f (0xedaa bytes) 000219 1786802845.214968 MIRROR_RESYNC 3 mirrors 000221 1786802846.024953 mapwrite 0x00b5e7 thru 0x010d73 (0x578d bytes) 000223 1786802846.330962 write 0x00a48e thru 0x019f84 (0xfaf7 bytes) 000225 1786802846.339630 read 0x02b05a thru 0x032a30 (0x79d7 bytes) 000229 1786802846.357513 write 0x033b7c thru 0x03c43c (0x88c1 bytes) 000230 1786802846.371663 mapwrite 0x017ee4 thru 0x024ce4 (0xce01 bytes) 000231 1786802846.462229 mapwrite 0x00e075 thru 0x01bce7 (0xdc73 bytes) 000234 1786802846.610064 mapread 0x03b8d7 thru 0x03ffff (0x4729 bytes) 000235 1786802846.627980 read 0x02c345 thru 0x03305a (0x6d16 bytes) 000240 1786802846.642146 mapread 0x00e642 thru 0x019d47 (0xb706 bytes) 000241 1786802846.645307 mapread 0x013b49 thru 0x023339 (0xf7f1 bytes) 000244 1786802846.649884 read 0x02985a thru 0x0360f4 (0xc89b bytes) 000245 1786802846.653138 write 0x00268a thru 0x005ed0 (0x3847 bytes) 000249 1786802846.673480 read 0x013434 thru 0x020f5a (0xdb27 bytes) 000252 1786802846.677432 read 0x00921d thru 0x00f7e7 (0x65cb bytes) 000253 1786802846.678903 trunc from 0x040000 to 0x0215b1 (0x1ea4e bytes) 000254 1786802846.774577 MIRROR_RESYNC 3 mirrors 000256 1786802847.690927 read 0x00ac54 thru 0x00d5fc (0x29a9 bytes) 000257 1786802847.725641 read 0x01d0c6 thru 0x02042b (0x3366 bytes) 000258 1786802847.753221 write 0x020f70 thru 0x02c4b6 (0xb547 bytes) 000259 1786802847.897574 mapwrite 0x007db4 thru 0x009deb (0x2038 bytes) 000261 1786802847.978227 mapwrite 0x03f19e thru 0x03ffff (0xe62 bytes) 000262 1786802848.071649 write 0x010130 thru 0x010a69 (0x93a bytes) 000263 1786802848.078785 write 0x0048da thru 0x00bb34 (0x725b bytes) 000265 1786802848.090082 trunc from 0x040000 to 0x031249 (0xedb6 bytes) 000266 1786802848.109274 mapwrite 0x015b82 thru 0x016497 (0x916 bytes) 000267 1786802848.168735 mapread 0x00a556 thru 0x00fc7b (0x5726 bytes) 000268 1786802848.190630 write 0x01a3d7 thru 0x0279ea (0xd614 bytes) 000271 1786802848.199527 write 0x02896e thru 0x030f3b (0x85ce bytes) 000272 1786802848.201797 read 0x030a33 thru 0x031249 (0x817 bytes) 000273 1786802848.202944 read 0x002087 thru 0x004292 (0x220c bytes) 000275 1786802848.219175 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3124a): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3124a): Mirror 7: 0x539d0eaa (stale) Mirror 8: 0x539d0eaa (stale) CRC-32 checksum value for chunk [0, 0x3124a): Mirror 1: 0x7dce5e1a Mirror 9: 0x7dce5e1a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3124a: skipped 000280 1786802849.441069 read 0x02a675 thru 0x031249 (0x6bd5 bytes) 000281 1786802849.462740 mapwrite 0x00fefd thru 0x01ec1b (0xed1f bytes) 000282 1786802849.709160 trunc from 0x03124a to 0x01d6f8 (0x13b51 bytes) 000283 1786802849.797525 mapread 0x01430c thru 0x01aea4 (0x6b99 bytes) 000284 1786802849.804076 mapread 0x016349 thru 0x01a9d7 (0x468f bytes) 000286 1786802849.816493 mapread 0x01c699 thru 0x01d6f8 (0x1060 bytes) 000288 1786802849.824642 read 0x019344 thru 0x01d6f8 (0x43b5 bytes) 000289 1786802849.826291 trunc from 0x01d6f9 to 0x001d5e (0x1b99a bytes) 000291 1786802849.869927 MIRROR_RESYNC 4 mirrors 000292 1786802850.636360 read 0x000bfd thru 0x001d5e (0x1162 bytes) 000293 1786802850.641108 trunc from 0x001d5f to 0x00d012 (0xb2b4 bytes) 000294 1786802850.741991 mapwrite 0x004700 thru 0x00ae37 (0x6738 bytes) 000295 1786802850.874893 MIRROR_RESYNC 4 mirrors 000298 1786802851.846390 mapread 0x000da2 thru 0x00d012 (0xc271 bytes) 000299 1786802851.873842 MIRROR_EXTEND to 5 mirrors 000300 1786802852.498801 read 0x00ce90 thru 0x00d012 (0x183 bytes) 000301 1786802852.504231 mapread 0x003dab thru 0x00504b (0x12a1 bytes) 000302 1786802852.524960 write 0x006a69 thru 0x011d88 (0xb320 bytes) 000306 1786802852.595647 mapread 0x00b7d8 thru 0x00dcea (0x2513 bytes) 000311 1786802852.599544 mapwrite 0x019a01 thru 0x023cec (0xa2ec bytes) 000313 1786802852.719554 trunc from 0x023ced to 0x01c73a (0x75b2 bytes) 000314 1786802852.746155 read 0x016fde thru 0x01c73a (0x575d bytes) 000315 1786802852.748215 write 0x01d08f thru 0x023cd4 (0x6c46 bytes) 000316 1786802852.749923 trunc from 0x023cd5 to 0x02d14f (0x947b bytes) truncating to largest ever: 0x3dd28 000317 1786802852.767689 trunc from 0x02d150 to 0x03dd27 (0x10bd8 bytes) 000318 1786802852.797830 MIRROR_RESYNC 5 mirrors 000325 1786802853.911835 write 0x01ea4e thru 0x0228ec (0x3e9f bytes) 000327 1786802854.036967 write 0x01ae0c thru 0x02485e (0x9a53 bytes) 000329 1786802854.056903 mapwrite 0x039ad8 thru 0x03ffff (0x6528 bytes) 000330 1786802854.206656 trunc from 0x040000 to 0x026734 (0x198cb bytes) 000332 1786802854.224405 write 0x02c650 thru 0x032817 (0x61c8 bytes) 000333 1786802854.226785 mapwrite 0x027cf6 thru 0x0352d8 (0xd5e3 bytes) 000334 1786802854.302183 read 0x02b5b3 thru 0x0323ce (0x6e1c bytes) 000338 1786802854.305950 write 0x03e94e thru 0x03f613 (0xcc6 bytes) 000340 1786802854.314264 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f614): Mirror 8: 0x46346cb0 (stale) Mirror 9: 0x46346cb0 (stale) Mirror 10: 0x46346cb0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f614): Mirror 8: 0x46346cb0 (stale) Mirror 9: 0x46346cb0 (stale) Mirror 10: 0x46346cb0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f614: skipped 000342 1786802854.770241 mapread 0x01b481 thru 0x022288 (0x6e08 bytes) 000343 1786802854.826595 MIRROR_RESYNC 4 mirrors 000344 1786802855.762260 MIRROR_EXTEND to 5 mirrors 000345 1786802856.348889 read 0x025f92 thru 0x02b756 (0x57c5 bytes) 000346 1786802856.359202 read 0x0161f5 thru 0x021ec0 (0xbccc bytes) 000347 1786802856.375516 read 0x00d41b thru 0x01b89d (0xe483 bytes) 000349 1786802856.376291 mapread 0x019bdf thru 0x027a40 (0xde62 bytes) 000350 1786802856.380040 mapwrite 0x032e2a thru 0x037022 (0x41f9 bytes) 000353 1786802856.551046 read 0x039e77 thru 0x03e752 (0x48dc bytes) 000355 1786802856.554605 mapread 0x002f32 thru 0x011de8 (0xeeb7 bytes) 000356 1786802856.557587 write 0x00bde9 thru 0x00ead1 (0x2ce9 bytes) 000358 1786802856.559557 mapread 0x021c42 thru 0x02f7a0 (0xdb5f bytes) 000359 1786802856.561526 trunc from 0x03f614 to 0x0019f2 (0x3dc21 bytes) 000361 1786802856.585612 write 0x02ebf3 thru 0x03af58 (0xc366 bytes) 000363 1786802856.600249 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3af59): Mirror 8: 0xbb59d9f1 (stale) Mirror 9: 0xbb59d9f1 (stale) Mirror 10: 0xbb59d9f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3af59): Mirror 8: 0xbb59d9f1 (stale) Mirror 9: 0xbb59d9f1 (stale) Mirror 10: 0xbb59d9f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3af59: skipped 000364 1786802856.951207 trunc from 0x03af59 to 0x01aca2 (0x202b6 bytes) 000365 1786802856.974827 MIRROR_RESYNC 4 mirrors 000366 1786802857.963324 MIRROR_EXTEND to 5 mirrors 000367 1786802858.635812 mapread 0x0092cf thru 0x00ea73 (0x57a5 bytes) 000368 1786802858.659931 read 0x001f65 thru 0x00bd0b (0x9da7 bytes) 000369 1786802858.665262 mapwrite 0x019cd3 thru 0x027741 (0xda6f bytes) 000371 1786802858.910272 mapread 0x004252 thru 0x008637 (0x43e6 bytes) 000372 1786802858.922067 read 0x0123b7 thru 0x01f883 (0xd4cd bytes) 000374 1786802858.923507 trunc from 0x027742 to 0x012367 (0x153da bytes) 000376 1786802858.967447 MIRROR_RESYNC 5 mirrors 000377 1786802860.039968 read 0x00dee0 thru 0x010380 (0x24a1 bytes) 000378 1786802860.050218 trunc from 0x012368 to 0x032103 (0x1fd9c bytes) 000379 1786802860.156325 write 0x020a62 thru 0x030033 (0xf5d2 bytes) 000380 1786802860.179860 read 0x0080dc thru 0x00feb7 (0x7ddc bytes) 000382 1786802860.194416 mapread 0x0202f4 thru 0x02060c (0x319 bytes) 000384 1786802860.202091 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32104): Mirror 9: 0x2b085c22 (stale) Mirror 10: 0x2b085c22 (stale) Mirror 11: 0x2b085c22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32104): Mirror 9: 0x2b085c22 (stale) Mirror 10: 0x2b085c22 (stale) Mirror 11: 0x2b085c22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32104: skipped 000385 1786802860.602199 trunc from 0x032104 to 0x009446 (0x28cbd bytes) 000386 1786802860.635831 write 0x013639 thru 0x01c0e2 (0x8aaa bytes) 000387 1786802860.639808 mapwrite 0x01ff20 thru 0x027ac8 (0x7ba9 bytes) 000388 1786802860.746956 read 0x00e2b7 thru 0x01611b (0x7e65 bytes) 000390 1786802860.757844 trunc from 0x027ac9 to 0x00fe06 (0x17cc2 bytes) 000395 1786802860.780412 read 0x003b42 thru 0x00e02b (0xa4ea bytes) 000397 1786802860.792088 write 0x038b21 thru 0x03ffff (0x74df bytes) 000398 1786802860.798010 mapread 0x00b26c thru 0x012a9f (0x7834 bytes) 000399 1786802860.812890 read 0x0286d4 thru 0x02ac77 (0x25a4 bytes) 000400 1786802860.814094 trunc from 0x040000 to 0x0232cf (0x1cd30 bytes) 000402 1786802860.848115 mapread 0x012f3f thru 0x01a950 (0x7a12 bytes) 000403 1786802860.849669 trunc from 0x0232d0 to 0x01a51f (0x8db0 bytes) 000404 1786802860.878079 MIRROR_RESYNC 4 mirrors 000405 1786802861.920519 mapread 0x00242b thru 0x008e9d (0x6a73 bytes) 000406 1786802861.949053 mapread 0x00ab1f thru 0x0171fc (0xc6de bytes) 000407 1786802861.958757 trunc from 0x01a520 to 0x02454f (0xa030 bytes) 000408 1786802862.061111 read 0x018010 thru 0x01d79e (0x578f bytes) 000411 1786802862.090279 read 0x016e6f thru 0x01e676 (0x7808 bytes) 000412 1786802862.103848 mapread 0x0200ae thru 0x02454f (0x44a2 bytes) 000413 1786802862.105816 mapread 0x017c19 thru 0x020ee5 (0x92cd bytes) 000414 1786802862.114299 MIRROR_RESYNC 4 mirrors 000415 1786802862.941441 mapwrite 0x01a556 thru 0x021f4f (0x79fa bytes) 000416 1786802863.212223 mapread 0x0125fd thru 0x01d10c (0xab10 bytes) 000418 1786802863.231924 read 0x00e278 thru 0x018b2e (0xa8b7 bytes) 000419 1786802863.237488 read 0x00b928 thru 0x017537 (0xbc10 bytes) 000422 1786802863.243892 mapread 0x003df6 thru 0x00d178 (0x9383 bytes) 000423 1786802863.265003 mapread 0x0206ac thru 0x02454f (0x3ea4 bytes) 000425 1786802863.274832 mapwrite 0x026a90 thru 0x02b5d1 (0x4b42 bytes) 000428 1786802863.417282 read 0x0187f0 thru 0x01d9e8 (0x51f9 bytes) 000429 1786802863.420466 mapread 0x00b8a3 thru 0x00cf64 (0x16c2 bytes) 000430 1786802863.422618 trunc from 0x02b5d2 to 0x036348 (0xad77 bytes) 000434 1786802863.463420 mapwrite 0x026de5 thru 0x0306ca (0x98e6 bytes) 000436 1786802863.600044 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36349): Mirror 10: 0x1b5c85d4 (stale) Mirror 11: 0x1b5c85d4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36349): Mirror 10: 0x1b5c85d4 (stale) Mirror 11: 0x1b5c85d4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36349: skipped 000437 1786802864.075773 mapread 0x02b538 thru 0x03182d (0x62f6 bytes) 000438 1786802864.115125 write 0x02228d thru 0x02b02e (0x8da2 bytes) 000439 1786802864.133900 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36349): Mirror 11: 0x1b5c85d4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36349): Mirror 11: 0x1b5c85d4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36349: skipped 000440 1786802864.507219 trunc from 0x036349 to 0x024367 (0x11fe1 bytes) 000442 1786802864.527193 read 0x0208b6 thru 0x024367 (0x3ab2 bytes) 000443 1786802864.536766 trunc from 0x024368 to 0x02bf43 (0x7bdc bytes) 000446 1786802864.561021 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) 000448 1786802864.693285 trunc from 0x02bf44 to 0x01714c (0x14df7 bytes) 000449 1786802864.713698 mapread 0x004212 thru 0x013b20 (0xf90f bytes) 000450 1786802864.741024 write 0x030024 thru 0x034618 (0x45f5 bytes) 000451 1786802864.747183 MIRROR_EXTEND to 2 mirrors 000453 1786802865.468828 write 0x034956 thru 0x038922 (0x3fcd bytes) 000454 1786802865.542922 write 0x001532 thru 0x0106b7 (0xf186 bytes) 000457 1786802865.555890 write 0x0317f6 thru 0x0336b6 (0x1ec1 bytes) 000458 1786802865.566566 mapread 0x0333cb thru 0x038922 (0x5558 bytes) 000460 1786802865.570882 write 0x03abd9 thru 0x03ff58 (0x5380 bytes) 000461 1786802865.577923 MIRROR_RESYNC 2 mirrors 000462 1786802866.288786 write 0x03c102 thru 0x03ec2e (0x2b2d bytes) 000463 1786802866.367604 mapwrite 0x0228ab thru 0x02a172 (0x78c8 bytes) 000466 1786802866.443881 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ff59): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ff59): Mirror 2: 0x80b3d1bc (stale) CRC-32 checksum value for chunk [0, 0x3ff59): Mirror 1: 0x9a984c89 Mirror 3: 0x9a984c89 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ff59: skipped 000467 1786802867.161938 read 0x00d421 thru 0x01831f (0xaeff bytes) 000468 1786802867.177289 read 0x028c85 thru 0x0350cc (0xc448 bytes) 000472 1786802867.192759 mapread 0x02e7e1 thru 0x030503 (0x1d23 bytes) 000473 1786802867.194797 mapwrite 0x02825c thru 0x02f161 (0x6f06 bytes) 000474 1786802867.453954 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3ff59): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ff59): Mirror 2: 0x80b3d1bc (stale) Mirror 3: 0x9a984c89 (stale) CRC-32 checksum value for chunk [0, 0x3ff59): Mirror 1: 0xf4538545 Mirror 4: 0xf4538545 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ff59: skipped 000475 1786802868.368138 mapwrite 0x02b479 thru 0x030c43 (0x57cb bytes) 000477 1786802868.607588 read 0x02b7e8 thru 0x035378 (0x9b91 bytes) 000478 1786802868.616702 trunc from 0x03ff59 to 0x01708c (0x28ecc bytes) 000480 1786802868.644847 mapwrite 0x03671a thru 0x039b3b (0x3422 bytes) 000481 1786802868.771276 mapread 0x036474 thru 0x039b3b (0x36c8 bytes) 000482 1786802868.773111 write 0x008037 thru 0x0160fa (0xe0c4 bytes) 000487 1786802868.826971 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39b3c): Mirror 2: 0xa87de2ff (stale) Mirror 4: 0xdaf45a9d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39b3c): Mirror 2: 0xa87de2ff (stale) Mirror 4: 0xdaf45a9d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39b3c: skipped 000493 1786802869.242418 read 0x00c363 thru 0x01adaa (0xea48 bytes) 000494 1786802869.264762 mapread 0x01556b thru 0x02205a (0xcaf0 bytes) 000495 1786802869.306217 mapwrite 0x02a6ec thru 0x02e723 (0x4038 bytes) 000496 1786802869.399941 read 0x038125 thru 0x039b3b (0x1a17 bytes) 000497 1786802869.414154 read 0x03788b thru 0x039b3b (0x22b1 bytes) 000498 1786802869.415489 trunc from 0x039b3c to 0x00d91d (0x2c21e bytes) 000499 1786802869.432935 trunc from 0x00d91e to 0x03cd75 (0x2f458 bytes) 000502 1786802869.456657 mapwrite 0x038869 thru 0x038c65 (0x3fd bytes) 000503 1786802869.556798 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3cd76): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd76): Mirror 2: 0xb94353f2 (stale) Mirror 4: 0x63fd606d (stale) CRC-32 checksum value for chunk [0, 0x3cd76): Mirror 1: 0x5ffbf85a Mirror 5: 0x5ffbf85a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd76: skipped 000505 1786802870.402166 read 0x02804f thru 0x030d0d (0x8cbf bytes) 000506 1786802870.418706 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3cd76): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd76): Mirror 2: 0xb94353f2 (stale) Mirror 4: 0x63fd606d (stale) CRC-32 checksum value for chunk [0, 0x3cd76): Mirror 1: 0x5ffbf85a Mirror 5: 0x5ffbf85a Mirror 6: 0x5ffbf85a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd76: skipped 000507 1786802871.628504 mapread 0x00a910 thru 0x00f7dc (0x4ecd bytes) 000508 1786802871.664765 write 0x025b47 thru 0x02bfe5 (0x649f bytes) 000510 1786802871.791255 read 0x0164c4 thru 0x01c6fb (0x6238 bytes) 000514 1786802871.809965 mapwrite 0x01ba6d thru 0x022a60 (0x6ff4 bytes) 000515 1786802871.919922 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cd76): Mirror 2: 0xb94353f2 (stale) Mirror 4: 0x63fd606d (stale) Mirror 5: 0x5ffbf85a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd76): Mirror 2: 0xb94353f2 (stale) Mirror 4: 0x63fd606d (stale) Mirror 5: 0x5ffbf85a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd76: skipped 000517 1786802872.480856 write 0x00e02f thru 0x0100b2 (0x2084 bytes) 000518 1786802872.498568 mapread 0x026cae thru 0x034c82 (0xdfd5 bytes) 000523 1786802872.553222 write 0x0006a3 thru 0x003665 (0x2fc3 bytes) 000524 1786802872.564773 mapread 0x01ef1e thru 0x02134f (0x2432 bytes) 000526 1786802872.586252 mapwrite 0x01c515 thru 0x02a053 (0xdb3f bytes) 000529 1786802872.684071 mapwrite 0x032d1f thru 0x038bc9 (0x5eab bytes) 000530 1786802872.754993 mapread 0x033281 thru 0x033f65 (0xce5 bytes) 000534 1786802872.760316 write 0x00f310 thru 0x014206 (0x4ef7 bytes) 000535 1786802872.769735 mapread 0x00b454 thru 0x00e438 (0x2fe5 bytes) 000536 1786802872.793106 mapread 0x01524c thru 0x016c25 (0x19da bytes) 000537 1786802872.808643 mapread 0x02e484 thru 0x033725 (0x52a2 bytes) 000540 1786802872.822852 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cd76): Mirror 4: 0x63fd606d (stale) Mirror 5: 0x5ffbf85a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd76): Mirror 4: 0x63fd606d (stale) Mirror 5: 0x5ffbf85a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd76: skipped 000542 1786802873.311440 mapread 0x02f294 thru 0x03466e (0x53db bytes) 000544 1786802873.353205 write 0x02ff8f thru 0x037e8f (0x7f01 bytes) 000547 1786802873.362975 write 0x00e824 thru 0x01e194 (0xf971 bytes) 000548 1786802873.385101 write 0x02271f thru 0x02c0cf (0x99b1 bytes) 000549 1786802873.401809 mapwrite 0x01b776 thru 0x021dde (0x6669 bytes) 000551 1786802873.554166 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cd76): Mirror 5: 0x5ffbf85a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd76): Mirror 5: 0x5ffbf85a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd76: skipped 000552 1786802874.009944 write 0x030909 thru 0x034475 (0x3b6d bytes) 000553 1786802874.024009 read 0x01f0b1 thru 0x02c678 (0xd5c8 bytes) 000554 1786802874.042224 mapread 0x022900 thru 0x032353 (0xfa54 bytes) 000555 1786802874.098744 mapread 0x03562a thru 0x03c7d3 (0x71aa bytes) 000557 1786802874.128653 mapread 0x03be79 thru 0x03cd75 (0xefd bytes) 000558 1786802874.135706 MIRROR_RESYNC 2 mirrors 000559 1786802875.090219 mapread 0x01f51f thru 0x02904d (0x9b2f bytes) 000560 1786802875.126465 mapwrite 0x02d27a thru 0x034045 (0x6dcc bytes) 000561 1786802875.339268 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3cd76): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd76): Mirror 5: 0x5251745c (stale) CRC-32 checksum value for chunk [0, 0x3cd76): Mirror 1: 0x96b5477 Mirror 6: 0x96b5477 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd76: skipped 000563 1786802876.921523 trunc from 0x03cd76 to 0x018349 (0x24a2c bytes) 000566 1786802877.095633 mapwrite 0x0209d0 thru 0x02d081 (0xc6b2 bytes) 000567 1786802877.274612 mapread 0x01b82c thru 0x0242ca (0x8a9f bytes) 000568 1786802877.276791 mapread 0x011158 thru 0x013b4c (0x29f5 bytes) 000569 1786802877.286187 trunc from 0x02d082 to 0x03423e (0x71bd bytes) 000570 1786802877.300794 mapwrite 0x02724a thru 0x02d3fb (0x61b2 bytes) 000571 1786802877.369786 mapread 0x02f32d thru 0x03423e (0x4f12 bytes) 000574 1786802877.400962 mapwrite 0x015825 thru 0x018b53 (0x332f bytes) 000575 1786802877.487418 read 0x02cb4f thru 0x031765 (0x4c17 bytes) 000579 1786802877.515434 mapwrite 0x00522a thru 0x00ff0f (0xace6 bytes) 000580 1786802877.586265 mapwrite 0x03550d thru 0x03fa72 (0xa566 bytes) 000581 1786802877.707485 mapread 0x00ce34 thru 0x01a40d (0xd5da bytes) 000584 1786802877.717087 mapread 0x036dc7 thru 0x038fc0 (0x21fa bytes) 000586 1786802877.719801 read 0x03b3d8 thru 0x03fa72 (0x469b bytes) 000588 1786802877.721669 write 0x0110ec thru 0x01a266 (0x917b bytes) 000589 1786802877.727927 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fa73): Mirror 6: 0xc7ae0c6b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fa73): Mirror 6: 0xc7ae0c6b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fa73: skipped 000591 1786802878.207803 write 0x0053df thru 0x00714e (0x1d70 bytes) 000592 1786802878.223488 trunc from 0x03fa73 to 0x027220 (0x18852 bytes) 000594 1786802878.243014 trunc from 0x027221 to 0x01760d (0xfc13 bytes) 000596 1786802878.267959 trunc from 0x01760e to 0x018c51 (0x1644 bytes) 000597 1786802878.292985 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x18c52): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18c52): Mirror 6: 0xefec1463 (stale) CRC-32 checksum value for chunk [0, 0x18c52): Mirror 1: 0x457e43a5 Mirror 7: 0x457e43a5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18c52: skipped 000598 1786802879.419374 trunc from 0x018c52 to 0x02a6d4 (0x11a83 bytes) 000599 1786802879.513223 write 0x010f7f thru 0x01a376 (0x93f8 bytes) 000603 1786802879.532482 trunc from 0x02a6d5 to 0x003984 (0x26d50 bytes) 000607 1786802879.570731 trunc from 0x003985 to 0x01f91d (0x1bf99 bytes) 000608 1786802879.621462 read 0x00026f thru 0x0050ae (0x4e40 bytes) 000609 1786802879.632448 mapread 0x000148 thru 0x00c6b2 (0xc56b bytes) 000612 1786802879.677212 trunc from 0x01f91e to 0x02400e (0x46f1 bytes) 000613 1786802879.709695 mapwrite 0x029fca thru 0x03957e (0xf5b5 bytes) 000616 1786802879.824558 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3957f): Mirror 7: 0x819ae11 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3957f): Mirror 7: 0x819ae11 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3957f: skipped 000618 1786802880.141312 write 0x00043c thru 0x00d99e (0xd563 bytes) 000622 1786802880.167547 read 0x022d27 thru 0x03052b (0xd805 bytes) 000623 1786802880.215415 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) 000624 1786802880.514024 trunc from 0x03957f to 0x00d676 (0x2bf08 bytes) 000626 1786802880.579103 mapread 0x0067d1 thru 0x00d676 (0x6ea6 bytes) 000629 1786802880.613456 write 0x001d3e thru 0x007932 (0x5bf5 bytes) 000630 1786802880.617916 trunc from 0x00d677 to 0x026c4c (0x195d6 bytes) 000631 1786802880.661176 mapread 0x0047ac thru 0x0117d1 (0xd026 bytes) 000633 1786802880.693314 write 0x0004e0 thru 0x00aa89 (0xa5aa bytes) 000634 1786802880.702239 mapread 0x00895f thru 0x00b4e8 (0x2b8a bytes) 000636 1786802880.705323 write 0x0151fe thru 0x01f48e (0xa291 bytes) 000637 1786802880.715755 mapread 0x003e2a thru 0x00add0 (0x6fa7 bytes) 000638 1786802880.718801 write 0x01884c thru 0x024bde (0xc393 bytes) 000640 1786802880.730452 mapread 0x00ee97 thru 0x01681c (0x7986 bytes) 000643 1786802880.735619 write 0x00ea25 thru 0x010ea1 (0x247d bytes) 000645 1786802880.739719 read 0x003abe thru 0x0134ac (0xf9ef bytes) 000646 1786802880.740783 trunc from 0x026c4d to 0x00ccc3 (0x19f89 bytes) 000650 1786802880.771602 mapread 0x002585 thru 0x00ccc3 (0xa73f bytes) 000651 1786802880.774237 trunc from 0x00ccc4 to 0x0190a7 (0xc3e4 bytes) 000652 1786802880.798470 trunc from 0x0190a8 to 0x02061c (0x7575 bytes) 000653 1786802880.824762 mapwrite 0x03ce30 thru 0x03ffff (0x31d0 bytes) 000654 1786802880.961447 MIRROR_EXTEND to 2 mirrors 000655 1786802882.021523 mapwrite 0x0065ee thru 0x009ff8 (0x3a0b bytes) 000657 1786802882.322219 write 0x02a840 thru 0x03a280 (0xfa41 bytes) 000658 1786802882.335966 write 0x02f46a thru 0x036487 (0x701e bytes) 000664 1786802882.343466 write 0x01890f thru 0x01f583 (0x6c75 bytes) 000665 1786802882.360398 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x30e3512d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdff06bc2 Mirror 3: 0xdff06bc2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000669 1786802883.362258 trunc from 0x040000 to 0x00aecf (0x35130 bytes) 000670 1786802883.505426 MIRROR_RESYNC 3 mirrors 000671 1786802884.594928 mapread 0x009bf6 thru 0x00aecf (0x12da bytes) 000672 1786802884.621489 mapwrite 0x035211 thru 0x03ac31 (0x5a21 bytes) 000673 1786802884.863675 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3ac32): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ac32): Mirror 2: 0x388018bc (stale) Mirror 3: 0x388018bc (stale) CRC-32 checksum value for chunk [0, 0x3ac32): Mirror 1: 0x44766309 Mirror 4: 0x44766309 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ac32: skipped 000674 1786802885.979959 trunc from 0x03ac32 to 0x02791c (0x13315 bytes) 000675 1786802886.139982 trunc from 0x02791d to 0x03ab1e (0x13202 bytes) 000676 1786802886.170666 trunc from 0x03ab1f to 0x034436 (0x66e8 bytes) 000677 1786802886.202827 mapread 0x01e308 thru 0x028087 (0x9d80 bytes) 000678 1786802886.236704 read 0x010328 thru 0x017e48 (0x7b21 bytes) 000679 1786802886.250711 write 0x027451 thru 0x02bd2a (0x48da bytes) 000680 1786802886.256179 write 0x03b8a2 thru 0x03ffff (0x475e bytes) 000681 1786802886.258895 mapwrite 0x03ae20 thru 0x03ffff (0x51e0 bytes) 000683 1786802886.398205 mapwrite 0x02b1ed thru 0x0392b7 (0xe0cb bytes) 000685 1786802886.554404 write 0x01cd9c thru 0x026bcf (0x9e34 bytes) 000687 1786802886.564097 mapread 0x034d95 thru 0x038fc9 (0x4235 bytes) 000689 1786802886.565769 mapread 0x0059bf thru 0x00813c (0x277e bytes) 000690 1786802886.610068 MIRROR_RESYNC 4 mirrors 000691 1786802887.905672 mapwrite 0x031048 thru 0x03188a (0x843 bytes) 000696 1786802888.102697 write 0x032042 thru 0x034ba6 (0x2b65 bytes) 000698 1786802888.116329 read 0x03d620 thru 0x03ffff (0x29e0 bytes) 000700 1786802888.131466 mapread 0x023d25 thru 0x0305a2 (0xc87e bytes) 000703 1786802888.156271 write 0x0192ad thru 0x01a252 (0xfa6 bytes) 000705 1786802888.179226 mapread 0x02fd1d thru 0x03f6bd (0xf9a1 bytes) 000706 1786802888.196970 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x88064a1b (stale) Mirror 3: 0x88064a1b (stale) Mirror 4: 0x88064a1b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4c57e030 Mirror 5: 0x4c57e030 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000707 1786802889.208566 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x88064a1b (stale) Mirror 3: 0x88064a1b (stale) Mirror 4: 0x88064a1b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4c57e030 Mirror 5: 0x4c57e030 Mirror 6: 0x4c57e030 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000708 1786802890.242720 read 0x0066f8 thru 0x008fb5 (0x28be bytes) 000709 1786802890.249852 trunc from 0x040000 to 0x03206e (0xdf91 bytes) 000710 1786802890.340566 trunc from 0x03206f to 0x01ca55 (0x15619 bytes) truncating to largest ever: 0x3e707 000711 1786802890.368639 trunc from 0x01ca56 to 0x03e706 (0x21cb1 bytes) 000712 1786802890.380749 write 0x022cbf thru 0x02c433 (0x9775 bytes) 000713 1786802890.400579 MIRROR_RESYNC 6 mirrors 000716 1786802891.389159 mapread 0x0172a9 thru 0x01aefe (0x3c56 bytes) 000717 1786802891.409905 MIRROR_RESYNC 6 mirrors 000718 1786802891.963241 MIRROR_EXTEND to 7 mirrors 000723 1786802892.816572 mapwrite 0x02e6f0 thru 0x035b5b (0x746c bytes) 000725 1786802893.021960 mapread 0x01bdc9 thru 0x02523d (0x9475 bytes) 000726 1786802893.045276 mapread 0x02ff86 thru 0x03e706 (0xe781 bytes) 000729 1786802893.061285 read 0x00127a thru 0x00481f (0x35a6 bytes) 000731 1786802893.071570 mapread 0x0050d1 thru 0x00541a (0x34a bytes) 000733 1786802893.082301 mapwrite 0x00c017 thru 0x013891 (0x787b bytes) 000734 1786802893.160528 trunc from 0x03e707 to 0x03e1f2 (0x514 bytes) 000735 1786802893.198164 read 0x019b62 thru 0x028d67 (0xf206 bytes) 000736 1786802893.199311 mapwrite 0x01384a thru 0x01a664 (0x6e1b bytes) 000737 1786802893.310170 mapread 0x01d096 thru 0x02aae2 (0xda4d bytes) 000738 1786802893.311766 trunc from 0x03e1f3 to 0x00dff4 (0x301fe bytes) 000740 1786802893.343554 read 0x0083d7 thru 0x00dff4 (0x5c1e bytes) 000744 1786802893.354623 mapwrite 0x00a33e thru 0x00f06a (0x4d2d bytes) 000746 1786802893.463877 mapwrite 0x03917b thru 0x03ffff (0x6e85 bytes) 000747 1786802893.541087 trunc from 0x040000 to 0x03be4a (0x41b5 bytes) 000748 1786802893.565576 read 0x014c31 thru 0x016f34 (0x2304 bytes) 000749 1786802893.566471 read 0x008605 thru 0x0124a1 (0x9e9d bytes) 000750 1786802893.571939 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3be4b): Mirror 2: 0x5e5da00e (stale) Mirror 4: 0x5e5da00e (stale) Mirror 5: 0x5e5da00e (stale) Mirror 6: 0x5e5da00e (stale) Mirror 7: 0x5e5da00e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3be4b): Mirror 2: 0x5e5da00e (stale) Mirror 4: 0x5e5da00e (stale) Mirror 5: 0x5e5da00e (stale) Mirror 6: 0x5e5da00e (stale) Mirror 7: 0x5e5da00e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3be4b: skipped 000751 1786802894.015100 write 0x0375e4 thru 0x03ffff (0x8a1c bytes) 000754 1786802894.026879 write 0x0020d3 thru 0x0071ff (0x512d bytes) 000755 1786802894.044248 write 0x0069a9 thru 0x011b6c (0xb1c4 bytes) 000756 1786802894.055211 trunc from 0x040000 to 0x02320c (0x1cdf3 bytes) 000757 1786802894.079626 write 0x00ce93 thru 0x00d95c (0xaca bytes) 000759 1786802894.083609 trunc from 0x02320d to 0x007ade (0x1b72e bytes) 000760 1786802894.119267 MIRROR_RESYNC 6 mirrors 000761 1786802895.300227 mapwrite 0x000452 thru 0x0062a0 (0x5e4f bytes) 000762 1786802895.614877 trunc from 0x007adf to 0x028212 (0x20734 bytes) 000764 1786802895.649281 mapwrite 0x0291d1 thru 0x02ee7f (0x5caf bytes) 000769 1786802895.744656 trunc from 0x02ee80 to 0x028dbd (0x60c2 bytes) 000770 1786802895.767297 trunc from 0x028dbe to 0x01ddc6 (0xaff7 bytes) 000773 1786802895.790531 mapwrite 0x032fc2 thru 0x034759 (0x1798 bytes) 000775 1786802895.877009 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3475a): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3475a): Mirror 2: 0x45a87455 (stale) Mirror 4: 0x45a87455 (stale) Mirror 5: 0x45a87455 (stale) Mirror 6: 0x45a87455 (stale) Mirror 7: 0x45a87455 (stale) CRC-32 checksum value for chunk [0, 0x3475a): Mirror 1: 0x83f25869 Mirror 8: 0x83f25869 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3475a: skipped 000778 1786802896.855812 trunc from 0x03475a to 0x032c17 (0x1b42 bytes) 000779 1786802896.952115 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32c18): Mirror 2: 0xc9eeefd (stale) Mirror 4: 0xc9eeefd (stale) Mirror 5: 0xc9eeefd (stale) Mirror 6: 0xc9eeefd (stale) Mirror 7: 0xc9eeefd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32c18): Mirror 2: 0xc9eeefd (stale) Mirror 4: 0xc9eeefd (stale) Mirror 5: 0xc9eeefd (stale) Mirror 6: 0xc9eeefd (stale) Mirror 7: 0xc9eeefd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32c18: skipped 000780 1786802897.371525 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x32c18): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32c18): Mirror 2: 0xc9eeefd (stale) Mirror 4: 0xc9eeefd (stale) Mirror 5: 0xc9eeefd (stale) Mirror 6: 0xc9eeefd (stale) Mirror 7: 0xc9eeefd (stale) CRC-32 checksum value for chunk [0, 0x32c18): Mirror 1: 0xd644abf5 Mirror 8: 0xd644abf5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32c18: skipped 000781 1786802898.329790 mapwrite 0x0187a4 thru 0x01df84 (0x57e1 bytes) 000782 1786802898.567425 read 0x00f1e4 thru 0x01e227 (0xf044 bytes) 000783 1786802898.579428 mapwrite 0x01565a thru 0x01bf40 (0x68e7 bytes) 000784 1786802898.689848 write 0x010f28 thru 0x016dc3 (0x5e9c bytes) truncating to largest ever: 0x3f125 000785 1786802898.691159 trunc from 0x032c18 to 0x03f124 (0xc50d bytes) 000786 1786802898.736992 read 0x03d4e7 thru 0x03f124 (0x1c3e bytes) 000787 1786802898.760833 MIRROR_RESYNC 7 mirrors 000789 1786802899.961061 read 0x00a6a5 thru 0x011a2c (0x7388 bytes) 000790 1786802899.976313 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) 000792 1786802900.562549 mapread 0x036ae4 thru 0x03e99d (0x7eba bytes) 000793 1786802900.587704 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) 000795 1786802901.150198 write 0x0181ed thru 0x021550 (0x9364 bytes) 000797 1786802901.243842 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3f125): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f125): Mirror 2: 0x188685f5 (stale) Mirror 4: 0x188685f5 (stale) Mirror 6: 0x188685f5 (stale) Mirror 8: 0x188685f5 (stale) CRC-32 checksum value for chunk [0, 0x3f125): Mirror 1: 0x8a5e85f Mirror 9: 0x8a5e85f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f125: skipped 000798 1786802902.112930 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3f125): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f125): Mirror 2: 0x188685f5 (stale) Mirror 4: 0x188685f5 (stale) Mirror 6: 0x188685f5 (stale) Mirror 8: 0x188685f5 (stale) CRC-32 checksum value for chunk [0, 0x3f125): Mirror 1: 0x8a5e85f Mirror 9: 0x8a5e85f Mirror 10: 0x8a5e85f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f125: skipped 000800 1786802903.156707 read 0x00bed6 thru 0x00f48d (0x35b8 bytes) 000801 1786802903.178278 read 0x00a0b4 thru 0x013b69 (0x9ab6 bytes) 000803 1786802903.211136 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3f125): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f125): Mirror 2: 0x188685f5 (stale) Mirror 4: 0x188685f5 (stale) Mirror 6: 0x188685f5 (stale) Mirror 8: 0x188685f5 (stale) CRC-32 checksum value for chunk [0, 0x3f125): Mirror 1: 0x8a5e85f Mirror 9: 0x8a5e85f Mirror 10: 0x8a5e85f Mirror 11: 0x8a5e85f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f125: skipped 000805 1786802904.188864 mapwrite 0x0309b8 thru 0x03f52b (0xeb74 bytes) 000806 1786802904.521863 trunc from 0x03f52c to 0x010525 (0x2f006 bytes) 000808 1786802904.557875 trunc from 0x010526 to 0x00bc3d (0x48e8 bytes) 000811 1786802904.586246 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xbc3e): Mirror 2: 0x903c74a5 (stale) Mirror 4: 0x903c74a5 (stale) Mirror 6: 0x903c74a5 (stale) Mirror 8: 0x903c74a5 (stale) Mirror 10: 0x903c74a5 (stale) Mirror 11: 0x903c74a5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbc3e): Mirror 2: 0x903c74a5 (stale) Mirror 4: 0x903c74a5 (stale) Mirror 6: 0x903c74a5 (stale) Mirror 8: 0x903c74a5 (stale) Mirror 10: 0x903c74a5 (stale) Mirror 11: 0x903c74a5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbc3e: skipped 000815 1786802905.091456 mapread 0x0014a2 thru 0x00884d (0x73ac bytes) 000816 1786802905.118744 mapwrite 0x03eb60 thru 0x03ffff (0x14a0 bytes) 000817 1786802905.227476 trunc from 0x040000 to 0x01b5ff (0x24a00 bytes) 000818 1786802905.253705 read 0x001d21 thru 0x00521e (0x34fe bytes) 000821 1786802905.260466 write 0x0251be thru 0x02b079 (0x5ebc bytes) 000823 1786802905.263584 mapwrite 0x03211b thru 0x034dfd (0x2ce3 bytes) 000824 1786802905.405051 mapread 0x011a83 thru 0x017e7d (0x63fb bytes) 000825 1786802905.417938 write 0x00257b thru 0x0087c2 (0x6248 bytes) 000826 1786802905.423160 mapwrite 0x019e8d thru 0x01a350 (0x4c4 bytes) 000828 1786802905.511534 mapread 0x00c000 thru 0x016d35 (0xad36 bytes) 000829 1786802905.518684 mapwrite 0x02cf22 thru 0x03a888 (0xd967 bytes) 000831 1786802905.607826 trunc from 0x03a889 to 0x0209fe (0x19e8a bytes) 000832 1786802905.630939 read 0x01570d thru 0x01e829 (0x911d bytes) 000835 1786802905.644131 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x209ff): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x209ff): Mirror 2: 0x3a40138d (stale) Mirror 4: 0x3a40138d (stale) Mirror 6: 0x3a40138d (stale) Mirror 8: 0x3a40138d (stale) Mirror 10: 0x611eafe3 (stale) Mirror 11: 0x611eafe3 (stale) CRC-32 checksum value for chunk [0, 0x209ff): Mirror 1: 0xcee5a50e Mirror 12: 0xcee5a50e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x209ff: skipped 000837 1786802906.700415 MIRROR_RESYNC 8 mirrors 000838 1786802907.922884 mapwrite 0x03c8a4 thru 0x03ffff (0x375c bytes) 000839 1786802908.131205 trunc from 0x040000 to 0x00e895 (0x3176a bytes) 000840 1786802908.150764 trunc from 0x00e896 to 0x03c8e2 (0x2e04d bytes) 000841 1786802908.169614 write 0x0098ff thru 0x014c01 (0xb303 bytes) 000842 1786802908.180887 read 0x011bd8 thru 0x013fc0 (0x23e9 bytes) 000843 1786802908.181787 mapread 0x032713 thru 0x03c8e2 (0xa1d0 bytes) 000844 1786802908.198298 mapwrite 0x01a1da thru 0x023655 (0x947c bytes) 000845 1786802908.275947 mapread 0x00d03b thru 0x01969c (0xc662 bytes) 000848 1786802908.279281 read 0x019947 thru 0x02334b (0x9a05 bytes) 000849 1786802908.286442 MIRROR_RESYNC 8 mirrors 000853 1786802909.787628 write 0x014777 thru 0x023967 (0xf1f1 bytes) 000854 1786802909.893934 write 0x0188fe thru 0x01d94f (0x5052 bytes) 000855 1786802909.896280 mapwrite 0x029028 thru 0x037d1f (0xecf8 bytes) 000856 1786802909.987497 write 0x00f7bc thru 0x017b09 (0x834e bytes) 000857 1786802909.994087 write 0x00ef18 thru 0x01d61c (0xe705 bytes) 000858 1786802910.001779 read 0x03882c thru 0x03c8e2 (0x40b7 bytes) 000859 1786802910.003166 read 0x03c40e thru 0x03c8e2 (0x4d5 bytes) 000861 1786802910.004721 trunc from 0x03c8e3 to 0x01680b (0x260d7 bytes) 000862 1786802910.029182 write 0x03f8fb thru 0x03ffff (0x705 bytes) 000864 1786802910.032933 write 0x03c6d4 thru 0x03ffff (0x392c bytes) 000865 1786802910.043236 trunc from 0x040000 to 0x00f9de (0x30621 bytes) 000867 1786802910.062612 trunc from 0x00f9df to 0x012a6c (0x308e bytes) 000872 1786802910.088768 MIRROR_RESYNC 8 mirrors 000874 1786802911.383932 write 0x00db69 thru 0x011c27 (0x40bf bytes) 000877 1786802911.505078 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12a6d): Mirror 2: 0xf566a673 (stale) Mirror 4: 0xf566a673 (stale) Mirror 6: 0xf566a673 (stale) Mirror 10: 0xf566a673 (stale) Mirror 11: 0xf566a673 (stale) Mirror 12: 0xf566a673 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12a6d): Mirror 2: 0xf566a673 (stale) Mirror 4: 0xf566a673 (stale) Mirror 6: 0xf566a673 (stale) Mirror 10: 0xf566a673 (stale) Mirror 11: 0xf566a673 (stale) Mirror 12: 0xf566a673 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12a6d: skipped 000878 1786802912.118059 read 0x005247 thru 0x00be30 (0x6bea bytes) 000879 1786802912.130115 mapwrite 0x00df55 thru 0x01b53e (0xd5ea bytes) 000881 1786802912.239965 trunc from 0x01b53f to 0x029f35 (0xe9f7 bytes) 000884 1786802912.261566 write 0x027ec2 thru 0x02f49e (0x75dd bytes) 000885 1786802912.270431 mapread 0x005192 thru 0x00a442 (0x52b1 bytes) 000887 1786802912.272262 read 0x0027c7 thru 0x003d22 (0x155c bytes) 000888 1786802912.281523 read 0x02ca10 thru 0x02f49e (0x2a8f bytes) 000889 1786802912.282457 mapwrite 0x0281a3 thru 0x037989 (0xf7e7 bytes) 000891 1786802912.357388 trunc from 0x03798a to 0x01db32 (0x19e57 bytes) 000892 1786802912.377258 trunc from 0x01db33 to 0x0166d8 (0x745a bytes) 000894 1786802912.401568 read 0x010f56 thru 0x0166d8 (0x5783 bytes) 000897 1786802912.425480 MIRROR_RESYNC 7 mirrors 000899 1786802913.715101 MIRROR_RESYNC 7 mirrors 000901 1786802914.142133 read 0x005726 thru 0x007c34 (0x250f bytes) 000902 1786802914.146729 mapwrite 0x00f9f9 thru 0x01c615 (0xcc1d bytes) 000904 1786802914.296565 write 0x03834a thru 0x03ffff (0x7cb6 bytes) 000905 1786802914.298595 write 0x025218 thru 0x0284e0 (0x32c9 bytes) 000906 1786802914.310249 mapread 0x01eb60 thru 0x0261ee (0x768f bytes) 000907 1786802914.318064 trunc from 0x040000 to 0x0364c0 (0x9b3f bytes) 000908 1786802914.336449 read 0x001071 thru 0x006db0 (0x5d40 bytes) 000910 1786802914.350220 mapread 0x028246 thru 0x0364c0 (0xe27b bytes) 000911 1786802914.354196 write 0x01502f thru 0x0211f4 (0xc1c6 bytes) 000912 1786802914.357697 mapread 0x000c02 thru 0x0028d7 (0x1cd6 bytes) 000913 1786802914.359492 trunc from 0x0364c1 to 0x0105c6 (0x25efa bytes) 000914 1786802914.383283 mapread 0x00f8ed thru 0x0105c6 (0xcda bytes) 000918 1786802914.389264 mapread 0x0057b5 thru 0x00f3dd (0x9c29 bytes) 000919 1786802914.395591 write 0x002de6 thru 0x00cf80 (0xa19b bytes) 000921 1786802914.399579 trunc from 0x0105c7 to 0x02c762 (0x1c19c bytes) 000925 1786802914.421842 mapwrite 0x02d8e9 thru 0x03a875 (0xcf8d bytes) 000926 1786802914.574125 write 0x007222 thru 0x0149e2 (0xd7c1 bytes) 000927 1786802914.586844 write 0x00900c thru 0x00a431 (0x1426 bytes) 000929 1786802914.592566 write 0x01453d thru 0x0203a5 (0xbe69 bytes) 000930 1786802914.605460 write 0x0192dd thru 0x028116 (0xee3a bytes) 000932 1786802914.615975 mapread 0x02e6cf thru 0x02fdb4 (0x16e6 bytes) 000933 1786802914.617983 write 0x00eda9 thru 0x010fe1 (0x2239 bytes) 000935 1786802914.624254 read 0x03a468 thru 0x03a875 (0x40e bytes) 000937 1786802914.626595 read 0x0189b5 thru 0x023647 (0xac93 bytes) 000939 1786802914.629134 mapread 0x00acb5 thru 0x01ac28 (0xff74 bytes) 000940 1786802914.640187 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a876): Mirror 2: 0x78051bb (stale) Mirror 6: 0x78051bb (stale) Mirror 10: 0x78051bb (stale) Mirror 11: 0x78051bb (stale) Mirror 12: 0x78051bb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a876): Mirror 2: 0x78051bb (stale) Mirror 6: 0x78051bb (stale) Mirror 10: 0x78051bb (stale) Mirror 11: 0x78051bb (stale) Mirror 12: 0x78051bb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a876: skipped 000941 1786802915.104589 trunc from 0x03a876 to 0x021065 (0x19810 bytes) 000942 1786802915.120273 write 0x01fc2d thru 0x02901f (0x93f3 bytes) 000943 1786802915.136577 read 0x021bb5 thru 0x025748 (0x3b94 bytes) 000944 1786802915.143321 write 0x0370af thru 0x0387c9 (0x171b bytes) 000945 1786802915.144849 trunc from 0x0387ca to 0x012445 (0x26384 bytes) 000946 1786802915.168530 read 0x002bb3 thru 0x006d37 (0x4185 bytes) 000947 1786802915.178307 mapwrite 0x006e6e thru 0x016aa3 (0xfc36 bytes) 000949 1786802915.254733 trunc from 0x016aa4 to 0x02cec4 (0x16421 bytes) 000950 1786802915.273823 write 0x01cb3f thru 0x01e78e (0x1c50 bytes) 000951 1786802915.291093 trunc from 0x02cec5 to 0x026de9 (0x60db bytes) 000952 1786802915.317441 write 0x03b849 thru 0x03ffff (0x47b7 bytes) 000953 1786802915.322396 mapwrite 0x016b30 thru 0x01f406 (0x88d7 bytes) 000958 1786802915.399157 mapread 0x014345 thru 0x01c038 (0x7cf4 bytes) 000962 1786802915.408758 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x9bacc4a0 (stale) Mirror 10: 0x9bacc4a0 (stale) Mirror 11: 0x9bacc4a0 (stale) Mirror 12: 0x9bacc4a0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x9bacc4a0 (stale) Mirror 10: 0x9bacc4a0 (stale) Mirror 11: 0x9bacc4a0 (stale) Mirror 12: 0x9bacc4a0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000964 1786802915.863643 mapwrite 0x00de4d thru 0x0107df (0x2993 bytes) 000966 1786802915.999955 mapwrite 0x010a46 thru 0x01fb73 (0xf12e bytes) 000975 1786802916.123742 write 0x01c5da thru 0x024047 (0x7a6e bytes) 000976 1786802916.135464 mapwrite 0x01302c thru 0x01e925 (0xb8fa bytes) 000977 1786802916.231157 trunc from 0x040000 to 0x00ac1f (0x353e0 bytes) 000979 1786802916.250948 mapwrite 0x006829 thru 0x00d4a5 (0x6c7d bytes) 000980 1786802916.370039 read 0x00465d thru 0x005548 (0xeec bytes) 000982 1786802916.371683 mapwrite 0x038f9f thru 0x03bfc7 (0x3029 bytes) 000983 1786802916.459815 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3bfc8): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bfc8): Mirror 6: 0x77c2f269 (stale) Mirror 10: 0x77c2f269 (stale) Mirror 11: 0x77c2f269 (stale) Mirror 12: 0x77c2f269 (stale) CRC-32 checksum value for chunk [0, 0x3bfc8): Mirror 1: 0x69885ee Mirror 13: 0x69885ee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bfc8: skipped 000984 1786802917.601602 read 0x001837 thru 0x00705e (0x5828 bytes) 000985 1786802917.616571 read 0x00742d thru 0x007ac3 (0x697 bytes) 000986 1786802917.627819 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x14f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bfc8): Mirror 6: 0x77c2f269 (stale) Mirror 10: 0x77c2f269 (stale) Mirror 11: 0x77c2f269 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bfc8): Mirror 6: 0x77c2f269 (stale) Mirror 10: 0x77c2f269 (stale) Mirror 11: 0x77c2f269 (stale) CRC-32 checksum value for chunk [0, 0x3bfc8): Mirror 1: 0x69885ee Mirror 13: 0x69885ee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bfc8: skipped 000987 1786802918.247544 MIRROR_RESYNC 5 mirrors 000988 1786802919.247597 read 0x015de1 thru 0x019461 (0x3681 bytes) 000990 1786802919.255241 mapread 0x0177c6 thru 0x0229cb (0xb206 bytes) 000992 1786802919.273878 read 0x01c274 thru 0x027a89 (0xb816 bytes) 000993 1786802919.294791 read 0x00c3b2 thru 0x00dc84 (0x18d3 bytes) 000996 1786802919.296179 mapread 0x00b669 thru 0x01475f (0x90f7 bytes) 000997 1786802919.297809 read 0x0376ee thru 0x03bfc7 (0x48da bytes) 000998 1786802919.298479 read 0x0147ba thru 0x01589b (0x10e2 bytes) All operations completed A-OK! PASS 70a (104s) == sanity-flr test 71: check mirror extend parent fid ==== 10:08:44 (1786802924) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 10:08:47 (1786802927) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.227618 s, 9.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199335 s, 5.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 10:08:54 (1786802934) Starting client: oleg603-client.virtnet: -o user_xattr,flock 192.168.206.103@tcp:/lustre /mnt/lustre2 Starting client: oleg603-client.virtnet: -o user_xattr,flock 192.168.206.103@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 7414368 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7406944 .. failed Extending file size to 8612096 .. Extending file size to 7825440 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8543008 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8957120 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8710240 .. Extending file size to 9050304 .. Extending file size to 8781024 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 59215 59217 59218 59220 59222 failed 192.168.206.103@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg603-client.virtnet /mnt/lustre2 (opts:) 192.168.206.103@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg603-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:10:17 (1786803017) Starting client: oleg603-client.virtnet: -o user_xattr,flock 192.168.206.103@tcp:/lustre /mnt/lustre2 Starting client: oleg603-client.virtnet: -o user_xattr,flock 192.168.206.103@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2056736 .. Extending file size to 5602400 .. Extending file size to 8706368 .. Extending file size to 8517568 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8839776 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8555552 .. Extending file size to 8890656 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9245376 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9019904 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 60810 60811 60812 60816 60817 done 192.168.206.103@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg603-client.virtnet /mnt/lustre2 (opts:) 192.168.206.103@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg603-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (83s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:11:40 (1786803100) Starting client: oleg603-client.virtnet: -o user_xattr,flock 192.168.206.103@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210560 2560 2205952 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 19456 3749888 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 25600 3743744 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 45056 7493632 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 2210560 2560 2205952 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3771392 19456 3749888 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 25600 3743744 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 45056 7493632 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124452 s, 16.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.438339 s, 4.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xcdc5d438 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.103@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg603-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:12:01 (1786803121) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118709 s, 17.7 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 10:12:09 (1786803129) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x58:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x129:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x129:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10a:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:12:15 (1786803135) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.170363 s, 12.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1eb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ec:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:12:23 (1786803143) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.212068 s, 9.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:12:31 (1786803151) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.343153 s, 6.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:13:04 (1786803184) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg603-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106766 s, 9.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } PASS 204d (58s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 10:14:02 (1786803242) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg603-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.180983 s, 11.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 10:15:00 (1786803300) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg603-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg603-server: Pool lustre.test_204f created Adding targets to pool oleg603-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg603-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.335177 s, 6.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg603-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg603-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg603-server: Pool lustre.test_204f destroyed PASS 204f (78s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 10:16:18 (1786803378) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 10:16:25 (1786803385) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 10:16:31 (1786803391) oleg603-server: Pool lustre.test_206 created Adding targets to pool oleg603-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg603-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg603-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg603-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg603-server: Pool lustre.test_206 destroyed PASS 206 (32s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 10:17:03 (1786803423) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910373 s, 11.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434879 s, 24.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x60d17dd8 Mirror 3: 0x60d17dd8 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 10:17:14 (1786803434) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 10:17:17 (1786803437) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 10:17:21 (1786803441) Starting client: oleg603-client.virtnet: -o user_xattr,flock 192.168.206.103@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.109275 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xb20000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1ef0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1ef0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1ef0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1b30000 Read this (4 bytes): BBBB 192.168.206.103@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg603-client.virtnet /mnt/lustre2 (opts:) PASS 209a (36s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 10:17:57 (1786803477) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0342101 s, 239 kB/s the very first read read with warm cache PASS 209b (10s) == sanity-flr test 210a: handle broken mirrored lovea ==== 10:18:07 (1786803487) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098423 s, 10.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7b lmm_fid: [0x200000405:0x7b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7b lmm_fid: [0x200000405:0x7b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11c:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611259 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-15 10:18:09.000000000 -0400 Modify: 2026-08-15 10:18:09.000000000 -0400 Change: 2026-08-15 10:18:11.000000000 -0400 Birth: 2026-08-15 10:18:09.000000000 -0400 PASS 210a (11s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 10:18:18 (1786803498) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 10:18:22 (1786803502) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.974857 s, 42.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0809632 s, 50.6 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 15 10:18 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x7d:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 15 10:18 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 10:18:34 (1786803514) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 10:18:43 (1786803523) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1676 sec =========== 10:18:53 (1786803533) === sanity-flr: start cleanup 10:18:55 (1786803535) === === sanity-flr: finish cleanup 10:19:03 (1786803543) ===