-----============= acceptance-small: sanity-flr ============----- Fri Aug 14 19:57:08 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg111-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 19:57:26 (1786751846) === oleg111-client.virtnet: executing check_config_client /mnt/lustre oleg111-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg111-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff999658a17800.idle_timeout=debug osc.lustre-OST0001-osc-ffff999658a17800.idle_timeout=debug disable quota as required oleg111-server: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 19:58:02 (1786751882) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7637] == sanity-flr test 0a: lfs mirror create with -N option == 19:58:05 (1786751885) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 19:58:24 (1786751904) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 19:58:27 (1786751907) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 19:58:31 (1786751911) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 19:58:59 (1786751939) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 19:59:03 (1786751943) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 19:59:07 (1786751947) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff999658a17800.stats=clear osc.lustre-OST0001-osc-ffff999658a17800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 19:59:16 (1786751956) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 19:59:27 (1786751967) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 19:59:35 (1786751975) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 19:59:38 (1786751978) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 19:59:47 (1786751987) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (7s) == sanity-flr test 2: create components from existing files ========================================================== 19:59:54 (1786751994) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:00:01 (1786752001) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:00:05 (1786752005) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:00:14 (1786752014) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:00:25 (1786752025) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:00:32 (1786752032) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.720208 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625134 s, 16.8 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:00:41 (1786752041) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0861715 s, 48.7 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 20:00:49 (1786752049) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:01:02 (1786752062) Stopping /mnt/lustre-ost1 (opts:) on oleg111-server oleg111-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff999658a17800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff999658a17800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg111-server: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg111-server: oleg111-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg111-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg111-server oleg111-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff999658a17800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff999658a17800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg111-server: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg111-server: oleg111-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg111-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (70s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:02:12 (1786752132) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.594026 s, 12.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg111-server oleg111-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff999658a17800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff999658a17800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg111-server: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg111-server: oleg111-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg111-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (46s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:02:58 (1786752178) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg111-server oleg111-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff999658a17800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff999658a17800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg111-server: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg111-server: oleg111-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg111-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg111-server oleg111-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff999658a17800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff999658a17800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg111-server: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg111-server: oleg111-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg111-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (77s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:04:15 (1786752255) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.4046 s, 11.9 MB/s -rw-r--r-- 1 root root 52428800 Aug 14 20:04 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.941s user 0m0.001s sys 0m0.169s Stopping /mnt/lustre-ost1 (opts:) on oleg111-server oleg111-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff999658a17800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff999658a17800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.025s user 0m0.000s sys 0m0.165s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg111-server: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg111-server: oleg111-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg111-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg111-server oleg111-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff999658a17800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff999658a17800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.398s user 0m0.001s sys 0m0.173s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg111-server: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg111-server: oleg111-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg111-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg111-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff999658a17800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff999658a17800.ost_server_uuid in FULL state after 0 sec PASS 33b (96s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:05:51 (1786752351) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:05:54 (1786752354) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:05:57 (1786752357) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 20:06:00 (1786752360) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 20:06:06 (1786752366) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.603964 s, 20.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:06:18 (1786752378) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029646 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309374 s, 33.9 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:06:27 (1786752387) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:06:37 (1786752397) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00266277 s, 385 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00340108 s, 301 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00116887 s, 876 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00172742 s, 593 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 20:06:49 (1786752409) create test files with size 15747409 .. d5712fa2e891d403d14be1d5a223bf1b - f55b4828a9243ee89651e16f1ad8f220 - f0d58132bd4b9bf9206285bb3723e29b - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 20:07:14 (1786752434) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:07:27 (1786752447) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:07:35 (1786752455) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103743 s, 10.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237633 s, 44.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 20:07:45 (1786752465) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118343 s, 17.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.298545 s, 14.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.218377 s, 9.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 20:08:04 (1786752484) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 20:08:07 (1786752487) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:08:11 (1786752491) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044975 s, 23.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 20:08:19 (1786752499) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:08:22 (1786752502) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:08:35 (1786752515) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.281562 s, 37.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:08:50 (1786752530) llite.lustre-ffff999658a97800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.292609 s, 35.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff999644a37000.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:09:07 (1786752547) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:09:48 (1786752588) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (40s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:10:28 (1786752628) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 20:10:33 (1786752633) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.239035 s, 13.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.324959 s, 9.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x89615ced (nosync,stale) Mirror 2: 0x89615ced (nosync,stale) ** verify mirror 2 doesn't change original checksum: 6ae36f7d817920c84830d3aacedd7ff6 - mirror 1 checksum: 86ab3b2fb820102c35cf32d458b2480f - mirror 2 checksum: 6ae36f7d817920c84830d3aacedd7ff6 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-14T20:10:37-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-14T20:10:37-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 6ae36f7d817920c84830d3aacedd7ff6 - mirror 1 checksum: 86ab3b2fb820102c35cf32d458b2480f - mirror 2 checksum: 86ab3b2fb820102c35cf32d458b2480f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-14T20:10:37-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-14T20:10:37-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } PASS 48 (15s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:10:48 (1786752648) Failing mds1 on oleg111-server Stopping /mnt/lustre-mds1 (opts:) on oleg111-server 20:10:55 (1786752655) shut down facet: mds1 facet_host: oleg111-server facet_failover_host: oleg111-server Failover mds1 to oleg111-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg111-server: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-MDT0000 20:11:30 (1786752690) targets are mounted 20:11:30 (1786752690) facet_failover done oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446618 s, 23.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 50A (63s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:11:51 (1786752711) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:11:54 (1786752714) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:11:57 (1786752717) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00719407 s, 2.3 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:12:05 (1786752725) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:12:08 (1786752728) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0117155 s, 350 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0516758 s, 79.3 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:12:15 (1786752735) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:12:37 (1786752757) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:13:05 (1786752785) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:13:31 (1786752811) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:13:56 (1786752836) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 20:14:03 (1786752843) mirror create pid 54342 mirror split pid 54343 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54342 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54343 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 20:15:11 (1786752911) Seed set to 2914 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1786752914.669276 mapwrite 0x006099 thru 0x0079b6 (0x191e bytes) 000003 1786752914.931729 MIRROR_EXTEND to 2 mirrors 000004 1786752915.783377 mapread 0x001a12 thru 0x0033bf (0x19ae bytes) 000005 1786752915.834691 write 0x00852e thru 0x00d6b0 (0x5183 bytes) 000007 1786752915.913011 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) truncating to largest ever: 0x1058c 000010 1786752916.099915 trunc from 0x00d6b1 to 0x01058b (0x2edb bytes) truncating to largest ever: 0x33fa1 000011 1786752916.119509 trunc from 0x01058c to 0x033fa0 (0x23a15 bytes) 000013 1786752916.142182 write 0x03c694 thru 0x03ffff (0x396c bytes) 000014 1786752916.144773 read 0x031163 thru 0x039362 (0x8200 bytes) 000015 1786752916.157337 MIRROR_EXTEND to 2 mirrors 000017 1786752916.930846 write 0x00c34c thru 0x012eb4 (0x6b69 bytes) 000018 1786752917.024163 write 0x001b40 thru 0x00b550 (0x9a11 bytes) 000019 1786752917.043444 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xaf3715fb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8e273a64 Mirror 3: 0x8e273a64 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000021 1786752917.991535 read 0x033d49 thru 0x03df95 (0xa24d bytes) 000023 1786752918.028702 read 0x018ac0 thru 0x023fff (0xb540 bytes) 000025 1786752918.050552 mapread 0x029405 thru 0x02f1e3 (0x5ddf bytes) truncating to largest ever: 0x37059 000026 1786752918.063810 trunc from 0x040000 to 0x037058 (0x8fa7 bytes) 000027 1786752918.139271 write 0x011916 thru 0x015cc1 (0x43ac bytes) 000029 1786752918.150288 mapwrite 0x02a161 thru 0x02d8c5 (0x3765 bytes) 000033 1786752918.215008 write 0x009361 thru 0x01122e (0x7ece bytes) 000034 1786752918.221434 trunc from 0x037059 to 0x027262 (0xfdf6 bytes) 000035 1786752918.268436 read 0x005f51 thru 0x0098bc (0x396c bytes) 000036 1786752918.284532 mapwrite 0x01fd95 thru 0x02371e (0x398a bytes) 000038 1786752918.371824 write 0x019c1d thru 0x01ad90 (0x1174 bytes) 000041 1786752918.378395 read 0x00126b thru 0x008171 (0x6f07 bytes) 000044 1786752918.385667 write 0x02dc86 thru 0x036f49 (0x92c4 bytes) 000047 1786752918.390639 read 0x00001e thru 0x009882 (0x9865 bytes) 000052 1786752918.399430 write 0x00c392 thru 0x010f6f (0x4bde bytes) 000053 1786752918.402338 write 0x012efd thru 0x01e841 (0xb945 bytes) 000054 1786752918.407504 mapread 0x019528 thru 0x01e58d (0x5066 bytes) 000055 1786752918.415157 write 0x02078d thru 0x02440c (0x3c80 bytes) 000056 1786752918.416661 trunc from 0x036f4a to 0x007933 (0x2f616 bytes) 000057 1786752918.470662 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x7934): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7934): Mirror 2: 0x7457b51b (stale) Mirror 3: 0xa80b2cef (stale) CRC-32 checksum value for chunk [0, 0x7934): Mirror 1: 0xa80b2cef Mirror 4: 0xa80b2cef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7934: skipped 000067 1786752919.612620 write 0x01d94d thru 0x02ba52 (0xe106 bytes) 000068 1786752919.688347 read 0x003170 thru 0x010368 (0xd1f9 bytes) 000071 1786752919.700938 read 0x003264 thru 0x0070c7 (0x3e64 bytes) 000072 1786752919.701715 write 0x0397f4 thru 0x03a5d6 (0xde3 bytes) 000073 1786752919.708377 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3a5d7): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a5d7): Mirror 2: 0x8842858c (stale) Mirror 3: 0xbb73d7a1 (stale) Mirror 4: 0x9e8d1995 (stale) CRC-32 checksum value for chunk [0, 0x3a5d7): Mirror 1: 0xa3cb6163 Mirror 5: 0xa3cb6163 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a5d7: skipped 000075 1786752920.585821 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3a5d7): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a5d7): Mirror 2: 0x8842858c (stale) Mirror 3: 0xbb73d7a1 (stale) Mirror 4: 0x9e8d1995 (stale) CRC-32 checksum value for chunk [0, 0x3a5d7): Mirror 1: 0xa3cb6163 Mirror 5: 0xa3cb6163 Mirror 6: 0xa3cb6163 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a5d7: skipped 000077 1786752921.528508 read 0x02f4ea thru 0x031979 (0x2490 bytes) 000078 1786752921.535094 write 0x02b230 thru 0x030df5 (0x5bc6 bytes) 000079 1786752921.610969 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a5d7): Mirror 2: 0x8842858c (stale) Mirror 4: 0x9e8d1995 (stale) Mirror 5: 0xa3cb6163 (stale) Mirror 6: 0xa3cb6163 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a5d7): Mirror 2: 0x8842858c (stale) Mirror 4: 0x9e8d1995 (stale) Mirror 5: 0xa3cb6163 (stale) Mirror 6: 0xa3cb6163 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a5d7: skipped 000080 1786752922.039193 mapwrite 0x02f15f thru 0x03929a (0xa13c bytes) 000083 1786752922.136052 write 0x011faa thru 0x020341 (0xe398 bytes) 000084 1786752922.150274 mapwrite 0x005173 thru 0x00f75d (0xa5eb bytes) 000087 1786752922.229223 mapwrite 0x02fa78 thru 0x0384cf (0x8a58 bytes) 000088 1786752922.284787 MIRROR_RESYNC 5 mirrors 000094 1786752924.367804 mapwrite 0x012e35 thru 0x01da79 (0xac45 bytes) 000095 1786752924.738270 trunc from 0x03a5d7 to 0x02d463 (0xd173 bytes) 000096 1786752924.793089 mapread 0x00fd15 thru 0x01b359 (0xb645 bytes) truncating to largest ever: 0x3eb3d 000097 1786752924.797258 trunc from 0x02d464 to 0x03eb3c (0x116d9 bytes) 000098 1786752924.844168 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3eb3d): Mirror 2: 0x4bcf77d2 (stale) Mirror 5: 0x4bcf77d2 (stale) Mirror 6: 0x4bcf77d2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eb3d): Mirror 2: 0x4bcf77d2 (stale) Mirror 5: 0x4bcf77d2 (stale) Mirror 6: 0x4bcf77d2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eb3d: skipped 000099 1786752925.441789 mapread 0x029a27 thru 0x03867d (0xec57 bytes) 000100 1786752925.465423 trunc from 0x03eb3d to 0x020edc (0x1dc60 bytes) 000102 1786752925.482897 mapwrite 0x01601f thru 0x01d4bb (0x749d bytes) 000103 1786752925.541320 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20edd): Mirror 2: 0xb8d7ecde (stale) Mirror 5: 0xb8d7ecde (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20edd): Mirror 2: 0xb8d7ecde (stale) Mirror 5: 0xb8d7ecde (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20edd: skipped 000104 1786752925.850936 read 0x000f0d thru 0x004c80 (0x3d74 bytes) 000105 1786752925.865445 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x20edd): Mirror 2: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20edd): Mirror 2: 0xb8d7ecde (stale) Mirror 5: 0xb8d7ecde (stale) CRC-32 checksum value for chunk [0, 0x20edd): Mirror 1: 0xe4f500a1 Mirror 6: 0xe4f500a1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20edd: skipped 000106 1786752926.907100 trunc from 0x020edd to 0x0292a5 (0x83c9 bytes) 000107 1786752927.064254 mapwrite 0x03cfcc thru 0x03ffff (0x3034 bytes) 000110 1786752927.337584 write 0x012809 thru 0x019275 (0x6a6d bytes) 000114 1786752927.394324 trunc from 0x040000 to 0x016675 (0x2998a bytes) 000115 1786752927.434238 trunc from 0x016676 to 0x02368f (0xd01a bytes) 000117 1786752927.483540 read 0x013096 thru 0x0208ee (0xd859 bytes) 000118 1786752927.511936 mapwrite 0x032787 thru 0x03ce95 (0xa70f bytes) 000119 1786752927.708536 MIRROR_RESYNC 4 mirrors 000120 1786752929.656877 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) 000122 1786752929.871173 read 0x010bc1 thru 0x01e955 (0xdd95 bytes) 000123 1786752929.901227 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) 000124 1786752930.180614 MIRROR_EXTEND to 3 mirrors 000125 1786752930.972440 mapwrite 0x004df6 thru 0x011040 (0xc24b bytes) 000128 1786752931.144979 mapread 0x02b1e8 thru 0x032ec6 (0x7cdf bytes) 000130 1786752931.153697 write 0x00d0d8 thru 0x00ec8c (0x1bb5 bytes) 000132 1786752931.155125 mapread 0x03a02e thru 0x03ce95 (0x2e68 bytes) 000135 1786752931.164868 trunc from 0x03ce96 to 0x009e5e (0x33037 bytes) 000139 1786752931.180614 write 0x01bd7d thru 0x01d6b1 (0x1935 bytes) 000140 1786752931.181626 read 0x00d925 thru 0x01cd6a (0xf446 bytes) 000142 1786752931.187823 write 0x03cf17 thru 0x03ffff (0x30e9 bytes) 000144 1786752931.189112 mapwrite 0x031501 thru 0x034555 (0x3055 bytes) 000146 1786752931.244392 trunc from 0x040000 to 0x01b284 (0x24d7b bytes) 000148 1786752931.259575 write 0x0019cd thru 0x00ad70 (0x93a4 bytes) 000149 1786752931.265982 write 0x01b334 thru 0x01cc8c (0x1959 bytes) 000150 1786752931.267195 read 0x002b52 thru 0x0072fa (0x47a9 bytes) 000151 1786752931.281714 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1cc8d): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cc8d): Mirror 5: 0x2a606566 (stale) Mirror 6: 0x2a606566 (stale) CRC-32 checksum value for chunk [0, 0x1cc8d): Mirror 1: 0x25d05f8f Mirror 7: 0x25d05f8f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cc8d: skipped 000153 1786752932.255347 write 0x03adcc thru 0x03ffff (0x5234 bytes) 000154 1786752932.322602 write 0x0086a5 thru 0x009886 (0x11e2 bytes) 000156 1786752932.336165 write 0x01fe2e thru 0x0241bb (0x438e bytes) 000158 1786752932.346383 trunc from 0x040000 to 0x01c7d8 (0x23827 bytes) 000160 1786752932.386410 trunc from 0x01c7d9 to 0x03bfc6 (0x1f7ee bytes) 000161 1786752932.410571 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bfc7): Mirror 6: 0xc990cda5 (stale) Mirror 7: 0xa7d6bfb5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bfc7): Mirror 6: 0xc990cda5 (stale) Mirror 7: 0xa7d6bfb5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bfc7: skipped 000163 1786752932.889269 read 0x00b8b4 thru 0x00c6ea (0xe37 bytes) 000165 1786752932.901315 mapwrite 0x008736 thru 0x00f62f (0x6efa bytes) 000168 1786752932.970062 mapwrite 0x03ea24 thru 0x03ffff (0x15dc bytes) 000171 1786752933.065100 mapread 0x032237 thru 0x03f794 (0xd55e bytes) 000172 1786752933.074798 read 0x025653 thru 0x02cb91 (0x753f bytes) 000175 1786752933.094703 read 0x02e369 thru 0x037a3e (0x96d6 bytes) 000176 1786752933.095447 write 0x0306fb thru 0x03959f (0x8ea5 bytes) 000177 1786752933.110286 trunc from 0x040000 to 0x0036bc (0x3c943 bytes) 000178 1786752933.152014 read 0x00291e thru 0x0036bc (0xd9f bytes) 000179 1786752933.157126 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36bd): Mirror 7: 0x1182eb5d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36bd): Mirror 7: 0x1182eb5d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36bd: skipped 000180 1786752933.570358 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) 000183 1786752933.739127 read 0x0031eb thru 0x0036bc (0x4d2 bytes) 000186 1786752933.765064 mapwrite 0x03e81d thru 0x03ffff (0x17e3 bytes) 000188 1786752933.873364 trunc from 0x040000 to 0x0196dc (0x26923 bytes) 000190 1786752933.907221 write 0x03fef4 thru 0x03ffff (0x10c bytes) 000192 1786752933.911183 write 0x02db59 thru 0x02f6c6 (0x1b6e bytes) 000193 1786752933.921147 write 0x013603 thru 0x0212c3 (0xdcc1 bytes) 000194 1786752933.940214 MIRROR_EXTEND to 2 mirrors 000195 1786752934.918974 mapwrite 0x01084f thru 0x011c41 (0x13f3 bytes) 000196 1786752935.032992 mapwrite 0x00fb04 thru 0x01e89c (0xed99 bytes) 000197 1786752935.084323 mapread 0x01bae4 thru 0x0279a4 (0xbec1 bytes) 000204 1786752935.094146 mapwrite 0x00f12a thru 0x0115da (0x24b1 bytes) 000205 1786752935.121126 write 0x003fc7 thru 0x00e8d6 (0xa910 bytes) 000206 1786752935.127353 trunc from 0x040000 to 0x006a7b (0x39584 bytes) 000208 1786752935.144479 trunc from 0x006a7c to 0x01762b (0x10bb0 bytes) 000209 1786752935.166354 mapread 0x01361f thru 0x01762b (0x400d bytes) 000211 1786752935.177786 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1762c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1762c): Mirror 2: 0x2951294e (stale) CRC-32 checksum value for chunk [0, 0x1762c): Mirror 1: 0x3b430443 Mirror 3: 0x3b430443 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1762c: skipped 000212 1786752935.872993 mapwrite 0x0172c1 thru 0x01f78b (0x84cb bytes) 000213 1786752936.039311 mapread 0x003967 thru 0x00894b (0x4fe5 bytes) 000214 1786752936.048377 mapread 0x01325b thru 0x01f78b (0xc531 bytes) 000216 1786752936.051176 write 0x02de4f thru 0x03bec3 (0xe075 bytes) 000217 1786752936.052873 trunc from 0x03bec4 to 0x036904 (0x55bf bytes) 000218 1786752936.071906 mapread 0x006cf1 thru 0x00cae6 (0x5df6 bytes) 000219 1786752936.074030 read 0x03109b thru 0x036904 (0x586a bytes) 000220 1786752936.075033 read 0x027681 thru 0x034379 (0xccf9 bytes) 000221 1786752936.082304 mapwrite 0x03dda4 thru 0x03ffff (0x225c bytes) 000222 1786752936.143818 mapread 0x011829 thru 0x012514 (0xcec bytes) 000223 1786752936.151567 MIRROR_RESYNC 3 mirrors 000227 1786752937.775636 write 0x038d92 thru 0x03ffd0 (0x723f bytes) 000229 1786752937.954138 read 0x01f197 thru 0x022ba4 (0x3a0e bytes) 000231 1786752937.964556 write 0x033b97 thru 0x03f522 (0xb98c bytes) 000232 1786752937.986851 read 0x00b4a8 thru 0x00d9e2 (0x253b bytes) 000233 1786752938.006807 MIRROR_RESYNC 3 mirrors 000234 1786752939.576961 read 0x00b79f thru 0x014e3f (0x96a1 bytes) 000236 1786752939.606104 read 0x018dfb thru 0x01ce7c (0x4082 bytes) 000237 1786752939.639771 trunc from 0x040000 to 0x015f58 (0x2a0a7 bytes) 000238 1786752939.823353 read 0x01079d thru 0x015f58 (0x57bc bytes) 000239 1786752939.860084 read 0x003d05 thru 0x00cc75 (0x8f71 bytes) 000240 1786752939.860973 write 0x02ae35 thru 0x02e683 (0x384f bytes) 000241 1786752939.863431 write 0x01b685 thru 0x029e5a (0xe7d6 bytes) 000242 1786752939.874393 mapwrite 0x02c714 thru 0x02d5cc (0xeb9 bytes) 000243 1786752940.014318 read 0x021e06 thru 0x02548d (0x3688 bytes) 000244 1786752940.015328 mapwrite 0x00536b thru 0x00b8c2 (0x6558 bytes) 000245 1786752940.182583 trunc from 0x02e684 to 0x00c937 (0x21d4c bytes) 000246 1786752940.267388 trunc from 0x00c938 to 0x01380a (0x6ed3 bytes) 000249 1786752940.317663 mapread 0x00420f thru 0x01380a (0xf5fc bytes) 000250 1786752940.364593 write 0x01c7cd thru 0x02c4b6 (0xfcea bytes) 000251 1786752940.390967 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2c4b7): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c4b7): Mirror 2: 0xbf88ca7a (stale) Mirror 3: 0xbf88ca7a (stale) CRC-32 checksum value for chunk [0, 0x2c4b7): Mirror 1: 0x27b16a0d Mirror 4: 0x27b16a0d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c4b7: skipped 000254 1786752942.173310 read 0x00d43a thru 0x00e3a8 (0xf6f bytes) 000257 1786752942.212395 trunc from 0x02c4b7 to 0x009f04 (0x225b2 bytes) 000258 1786752942.325493 mapread 0x0014af thru 0x009f04 (0x8a56 bytes) 000259 1786752942.352069 trunc from 0x009f05 to 0x02dd5b (0x23e57 bytes) 000260 1786752942.377408 mapread 0x01b92f thru 0x025df2 (0xa4c4 bytes) 000261 1786752942.409608 read 0x0066a1 thru 0x00d474 (0x6dd4 bytes) 000262 1786752942.425079 trunc from 0x02dd5c to 0x03220c (0x44b1 bytes) 000263 1786752942.454660 trunc from 0x03220d to 0x01c28e (0x15f7e bytes) 000265 1786752942.489150 mapread 0x00fec4 thru 0x016ac6 (0x6c03 bytes) 000266 1786752942.490953 mapwrite 0x036d3c thru 0x03ffff (0x92c4 bytes) 000267 1786752942.550706 trunc from 0x040000 to 0x007954 (0x386ab bytes) 000268 1786752942.564799 trunc from 0x007955 to 0x03b293 (0x3393f bytes) 000269 1786752942.588785 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3b294): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b294): Mirror 2: 0x29476437 (stale) Mirror 3: 0x29476437 (stale) Mirror 4: 0x6b86b3f3 (stale) CRC-32 checksum value for chunk [0, 0x3b294): Mirror 1: 0x78140752 Mirror 5: 0x78140752 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b294: skipped 000270 1786752944.098895 mapwrite 0x032b66 thru 0x03ffff (0xd49a bytes) 000272 1786752944.398802 write 0x019a02 thru 0x020fb9 (0x75b8 bytes) 000273 1786752944.415405 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd794172b (stale) Mirror 4: 0xd9874082 (stale) Mirror 5: 0xcfd19e3b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd794172b (stale) Mirror 4: 0xd9874082 (stale) Mirror 5: 0xcfd19e3b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000278 1786752945.064142 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xd9874082 (stale) Mirror 5: 0xcfd19e3b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xd9874082 (stale) Mirror 5: 0xcfd19e3b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000279 1786752945.444241 write 0x028c49 thru 0x0341fb (0xb5b3 bytes) 000282 1786752945.465937 mapwrite 0x02b6e6 thru 0x02ea30 (0x334b bytes) 000288 1786752945.575354 mapwrite 0x00fd1c thru 0x015734 (0x5a19 bytes) 000290 1786752945.734553 mapwrite 0x00e68c thru 0x00fd27 (0x169c bytes) 000291 1786752945.787014 trunc from 0x040000 to 0x029b1d (0x164e2 bytes) 000292 1786752945.831721 mapread 0x01e94a thru 0x0272d6 (0x898d bytes) 000294 1786752945.863150 write 0x02ba94 thru 0x03577c (0x9ce9 bytes) 000295 1786752945.865669 trunc from 0x03577d to 0x0291a5 (0xc5d7 bytes) 000298 1786752945.901378 trunc from 0x0291a6 to 0x027660 (0x1b45 bytes) 000299 1786752945.921273 mapread 0x0216df thru 0x027660 (0x5f82 bytes) 000300 1786752945.922345 mapread 0x01ba81 thru 0x0204af (0x4a2f bytes) 000302 1786752945.935037 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x27661): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27661): Mirror 4: 0xaa0f9264 (stale) Mirror 5: 0xee32df2e (stale) CRC-32 checksum value for chunk [0, 0x27661): Mirror 1: 0x2f24a6ef Mirror 6: 0x2f24a6ef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27661: skipped 000304 1786752947.299542 mapwrite 0x03f845 thru 0x03ffff (0x7bb bytes) 000306 1786752947.468156 MIRROR_RESYNC 4 mirrors 000307 1786752949.413425 read 0x0078cf thru 0x00a739 (0x2e6b bytes) 000308 1786752949.425269 read 0x03caa8 thru 0x03ffff (0x3558 bytes) 000310 1786752949.469994 write 0x00d64b thru 0x011a10 (0x43c6 bytes) 000311 1786752949.552663 read 0x021a6f thru 0x031647 (0xfbd9 bytes) 000312 1786752949.566028 write 0x00ae37 thru 0x015357 (0xa521 bytes) 000314 1786752949.574039 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x2c80fea0 (stale) Mirror 5: 0x2c80fea0 (stale) Mirror 6: 0x2c80fea0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf87b7432 Mirror 7: 0xf87b7432 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000315 1786752950.634286 read 0x013960 thru 0x017fae (0x464f bytes) 000316 1786752950.651290 mapread 0x038d76 thru 0x03ffff (0x728a bytes) 000317 1786752950.683794 read 0x007931 thru 0x00d8e6 (0x5fb6 bytes) 000321 1786752950.703887 MIRROR_RESYNC 5 mirrors 000322 1786752952.896868 read 0x0062ea thru 0x0136aa (0xd3c1 bytes) 000324 1786752952.926332 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) 000326 1786752953.407097 mapread 0x01cfb0 thru 0x025a22 (0x8a73 bytes) 000327 1786752953.445768 read 0x00b480 thru 0x0118fe (0x647f bytes) 000329 1786752953.493017 read 0x03aae5 thru 0x03ffff (0x551b bytes) 000330 1786752953.523350 mapwrite 0x00fb97 thru 0x013d0a (0x4174 bytes) 000332 1786752953.687174 mapwrite 0x007b3f thru 0x00e398 (0x685a bytes) 000335 1786752953.780161 write 0x009da7 thru 0x0171c4 (0xd41e bytes) 000336 1786752953.794093 write 0x0237bd thru 0x02515a (0x199e bytes) 000337 1786752953.816919 write 0x00ec06 thru 0x015f5c (0x7357 bytes) 000338 1786752953.826204 write 0x024608 thru 0x02c4f7 (0x7ef0 bytes) 000339 1786752953.846104 read 0x00450a thru 0x012906 (0xe3fd bytes) 000340 1786752953.863218 read 0x03fcd9 thru 0x03ffff (0x327 bytes) 000344 1786752953.866343 mapwrite 0x0091e3 thru 0x010fec (0x7e0a bytes) 000348 1786752953.921006 read 0x01f55f thru 0x0255b6 (0x6058 bytes) 000352 1786752953.930071 write 0x022f59 thru 0x030dab (0xde53 bytes) 000353 1786752953.932592 write 0x010a2c thru 0x015ef4 (0x54c9 bytes) 000354 1786752953.933924 trunc from 0x040000 to 0x01f81a (0x207e5 bytes) 000355 1786752953.980568 mapwrite 0x00c850 thru 0x01b257 (0xea08 bytes) 000356 1786752954.113608 read 0x00f242 thru 0x012e8d (0x3c4c bytes) 000357 1786752954.119252 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f81b): Mirror 6: 0x6f194c82 (stale) Mirror 7: 0x6f194c82 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f81b): Mirror 6: 0x6f194c82 (stale) Mirror 7: 0x6f194c82 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f81b: skipped 000358 1786752954.587107 write 0x017f07 thru 0x024d78 (0xce72 bytes) 000359 1786752954.592679 trunc from 0x024d79 to 0x011328 (0x13a50 bytes) 000362 1786752954.612044 read 0x0014a4 thru 0x0059f1 (0x454e bytes) 000363 1786752954.621436 mapwrite 0x001b5a thru 0x006988 (0x4e2f bytes) 000364 1786752954.676921 mapwrite 0x01c948 thru 0x027ac1 (0xb17a bytes) 000365 1786752954.751272 write 0x03a096 thru 0x03e494 (0x43ff bytes) 000366 1786752954.752560 trunc from 0x03e495 to 0x03d2d5 (0x11bf bytes) 000367 1786752954.772633 trunc from 0x03d2d6 to 0x00ce98 (0x3043d bytes) 000373 1786752954.793112 trunc from 0x00ce99 to 0x016851 (0x99b9 bytes) 000375 1786752954.811444 write 0x010e6c thru 0x019ea6 (0x903b bytes) 000376 1786752954.818073 write 0x004429 thru 0x007ea6 (0x3a7e bytes) 000377 1786752954.829083 MIRROR_RESYNC 3 mirrors 000379 1786752955.854057 read 0x000b28 thru 0x010415 (0xf8ee bytes) 000380 1786752955.871100 trunc from 0x019ea7 to 0x004d2c (0x1517a bytes) 000381 1786752955.960061 write 0x024af1 thru 0x02a3d5 (0x58e5 bytes) 000382 1786752955.966657 MIRROR_RESYNC 3 mirrors 000383 1786752957.100770 write 0x0309b0 thru 0x037c52 (0x72a3 bytes) 000384 1786752957.192660 read 0x013415 thru 0x01c900 (0x94ec bytes) 000385 1786752957.205711 mapread 0x020e38 thru 0x02b179 (0xa342 bytes) 000386 1786752957.242947 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37c53): Mirror 6: 0xf6d07697 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37c53): Mirror 6: 0xf6d07697 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37c53: skipped 000388 1786752957.542972 mapwrite 0x00a695 thru 0x010b9a (0x6506 bytes) 000389 1786752957.654592 read 0x01d3ed thru 0x021b82 (0x4796 bytes) 000393 1786752957.669846 mapread 0x00e1fc thru 0x019d60 (0xbb65 bytes) 000395 1786752957.684883 trunc from 0x037c53 to 0x039e38 (0x21e6 bytes) 000397 1786752957.717083 mapwrite 0x0281ef thru 0x0351c0 (0xcfd2 bytes) 000398 1786752957.860257 read 0x016c41 thru 0x017ff4 (0x13b4 bytes) 000402 1786752957.862586 mapwrite 0x0131c1 thru 0x015b1e (0x295e bytes) 000403 1786752957.951067 write 0x0077fd thru 0x00c124 (0x4928 bytes) 000407 1786752957.955977 trunc from 0x039e39 to 0x01c9f9 (0x1d43f bytes) 000408 1786752957.972059 trunc from 0x01c9fa to 0x020df5 (0x43fc bytes) 000410 1786752957.993571 trunc from 0x020df6 to 0x019665 (0x7790 bytes) 000412 1786752958.011924 write 0x039887 thru 0x03ffff (0x6779 bytes) 000416 1786752958.016958 mapread 0x01bd74 thru 0x029de7 (0xe074 bytes) 000417 1786752958.027011 mapread 0x038ffe thru 0x03f745 (0x6748 bytes) 000420 1786752958.035680 MIRROR_RESYNC 2 mirrors 000422 1786752958.944861 MIRROR_EXTEND to 3 mirrors 000424 1786752960.252928 write 0x00a76b thru 0x00fc3b (0x54d1 bytes) 000426 1786752960.328044 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x7d945626 (stale) Mirror 7: 0x7d945626 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4133b5f5 Mirror 8: 0x4133b5f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000427 1786752961.647989 trunc from 0x040000 to 0x038417 (0x7be8 bytes) 000428 1786752961.814262 trunc from 0x038418 to 0x01b877 (0x1cba0 bytes) 000430 1786752961.847890 trunc from 0x01b878 to 0x0116c2 (0xa1b5 bytes) 000432 1786752961.872912 trunc from 0x0116c3 to 0x002a9e (0xec24 bytes) 000434 1786752961.903242 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2a9f): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a9f): Mirror 6: 0xc2390547 (stale) Mirror 7: 0xc2390547 (stale) Mirror 8: 0xc2390547 (stale) CRC-32 checksum value for chunk [0, 0x2a9f): Mirror 1: 0xc2390547 Mirror 9: 0xc2390547 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a9f: skipped 000435 1786752963.000620 read 0x00080b thru 0x002a9e (0x2294 bytes) 000439 1786752963.010408 mapread 0x0013d7 thru 0x002a9e (0x16c8 bytes) 000440 1786752963.012028 mapwrite 0x02261d thru 0x02a95b (0x833f bytes) 000443 1786752963.289216 write 0x02c858 thru 0x03b23e (0xe9e7 bytes) 000446 1786752963.306762 write 0x0113a3 thru 0x0206ab (0xf309 bytes) 000447 1786752963.323424 write 0x01cc06 thru 0x021542 (0x493d bytes) 000448 1786752963.331772 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b23f): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b23f): Mirror 6: 0x68204b2e (stale) Mirror 7: 0x68204b2e (stale) Mirror 8: 0x71ddc1ec (stale) Mirror 9: 0x1181e514 (stale) CRC-32 checksum value for chunk [0, 0x3b23f): Mirror 1: 0x1adcfa98 Mirror 10: 0x1adcfa98 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b23f: skipped 000449 1786752964.584802 read 0x0042fb thru 0x009008 (0x4d0e bytes) 000451 1786752964.613046 write 0x03879f thru 0x03ffff (0x7861 bytes) 000452 1786752964.688450 trunc from 0x040000 to 0x0349b4 (0xb64b bytes) 000453 1786752964.716887 mapwrite 0x015701 thru 0x015977 (0x277 bytes) 000456 1786752964.764635 mapwrite 0x009092 thru 0x016fae (0xdf1d bytes) 000460 1786752964.836756 read 0x021e2a thru 0x0291e7 (0x73be bytes) 000461 1786752964.847014 mapwrite 0x03aea5 thru 0x03ffff (0x515b bytes) 000462 1786752964.905903 write 0x011d0e thru 0x017a46 (0x5d39 bytes) 000464 1786752964.907964 mapread 0x03fdd6 thru 0x03ffff (0x22a bytes) 000465 1786752964.913292 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x7d945626 (stale) Mirror 7: 0x7d945626 (stale) Mirror 8: 0x4133b5f5 (stale) Mirror 9: 0xc637eaf4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x7d945626 (stale) Mirror 7: 0x7d945626 (stale) Mirror 8: 0x4133b5f5 (stale) Mirror 9: 0xc637eaf4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000468 1786752965.375606 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x7d945626 (stale) Mirror 8: 0x4133b5f5 (stale) Mirror 9: 0xc637eaf4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x7d945626 (stale) Mirror 8: 0x4133b5f5 (stale) Mirror 9: 0xc637eaf4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000469 1786752965.778138 mapread 0x03e415 thru 0x03ffff (0x1beb bytes) 000470 1786752965.793928 read 0x0053f6 thru 0x00bfa3 (0x6bae bytes) truncating to largest ever: 0x3fc15 000474 1786752965.807255 trunc from 0x040000 to 0x03fc14 (0x3eb bytes) 000475 1786752965.840750 MIRROR_RESYNC 4 mirrors 000476 1786752967.953996 mapwrite 0x00bea9 thru 0x015ad7 (0x9c2f bytes) 000477 1786752968.115027 read 0x00781d thru 0x015ac8 (0xe2ac bytes) 000478 1786752968.115784 trunc from 0x03fc15 to 0x038944 (0x72d0 bytes) 000480 1786752968.133452 write 0x00f745 thru 0x01a9ff (0xb2bb bytes) 000482 1786752968.135686 write 0x038230 thru 0x03ffff (0x7dd0 bytes) 000484 1786752968.149572 MIRROR_RESYNC 4 mirrors 000485 1786752970.692859 mapread 0x01b793 thru 0x01da20 (0x228e bytes) 000487 1786752970.732831 trunc from 0x040000 to 0x021504 (0x1eafb bytes) 000488 1786752970.964779 mapread 0x006bad thru 0x00759f (0x9f3 bytes) 000489 1786752970.991526 write 0x00c496 thru 0x017e2d (0xb998 bytes) 000491 1786752971.019231 mapread 0x00a56b thru 0x00eeea (0x4980 bytes) 000492 1786752971.071000 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x21505): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21505): Mirror 6: 0xef8b45ef (stale) Mirror 8: 0xef8b45ef (stale) Mirror 9: 0xef8b45ef (stale) CRC-32 checksum value for chunk [0, 0x21505): Mirror 1: 0x64ceaa58 Mirror 10: 0x64ceaa58 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21505: skipped 000493 1786752972.237135 read 0x006f0d thru 0x0115ec (0xa6e0 bytes) 000494 1786752972.248826 mapwrite 0x011aca thru 0x016f68 (0x549f bytes) 000495 1786752972.423403 read 0x00bf48 thru 0x00e232 (0x22eb bytes) 000496 1786752972.441989 mapread 0x00c180 thru 0x011f6a (0x5deb bytes) 000497 1786752972.444131 trunc from 0x021505 to 0x009221 (0x182e3 bytes) 000498 1786752972.509694 write 0x00ecf7 thru 0x019b96 (0xaea0 bytes) 000500 1786752972.511905 write 0x03e988 thru 0x03ffff (0x1678 bytes) 000501 1786752972.528095 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8173fc67 (stale) Mirror 8: 0x8173fc67 (stale) Mirror 10: 0xf6bf00b5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8173fc67 (stale) Mirror 8: 0x8173fc67 (stale) Mirror 10: 0xf6bf00b5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000504 1786752972.853185 mapread 0x030042 thru 0x035836 (0x57f5 bytes) 000506 1786752972.876651 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8173fc67 (stale) Mirror 8: 0x8173fc67 (stale) Mirror 10: 0xf6bf00b5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2089f4db Mirror 11: 0x2089f4db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000508 1786752974.467912 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8173fc67 (stale) Mirror 10: 0xf6bf00b5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8173fc67 (stale) Mirror 10: 0xf6bf00b5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2089f4db Mirror 11: 0x2089f4db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000509 1786752975.450653 trunc from 0x040000 to 0x000319 (0x3fce6 bytes) 000510 1786752975.581535 mapwrite 0x027aea thru 0x02a257 (0x276e bytes) 000512 1786752975.786562 mapread 0x004b86 thru 0x00b802 (0x6c7d bytes) 000513 1786752975.815627 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a258): Mirror 10: 0x98bdc5a8 (stale) Mirror 11: 0x49524ddc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a258): Mirror 10: 0x98bdc5a8 (stale) Mirror 11: 0x49524ddc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a258: skipped 000515 1786752976.294885 trunc from 0x02a258 to 0x01957e (0x10cd9 bytes) 000516 1786752976.327136 trunc from 0x01957f to 0x00c2b0 (0xd2ce bytes) 000517 1786752976.352956 MIRROR_RESYNC 3 mirrors 000519 1786752976.970407 trunc from 0x00c2b1 to 0x01a5ca (0xe31a bytes) 000525 1786752977.020773 read 0x018f82 thru 0x01a5ca (0x1649 bytes) 000526 1786752977.026246 mapread 0x000e19 thru 0x004066 (0x324e bytes) 000527 1786752977.042002 read 0x00ccd2 thru 0x0102fa (0x3629 bytes) 000529 1786752977.043998 write 0x007f64 thru 0x012839 (0xa8d6 bytes) 000532 1786752977.049738 trunc from 0x01a5cb to 0x0059f3 (0x14bd7 bytes) 000533 1786752977.072558 read 0x002327 thru 0x0059f3 (0x36cd bytes) 000534 1786752977.073542 write 0x02dd69 thru 0x03404c (0x62e4 bytes) 000535 1786752977.075567 trunc from 0x03404d to 0x005378 (0x2ecd4 bytes) 000536 1786752977.087451 write 0x0202ba thru 0x027573 (0x72ba bytes) 000537 1786752977.088743 mapwrite 0x01da97 thru 0x01e5a8 (0xb12 bytes) 000538 1786752977.126130 write 0x00f864 thru 0x01cc9d (0xd43a bytes) 000539 1786752977.128925 mapread 0x00cdc4 thru 0x01885e (0xba9b bytes) 000540 1786752977.130319 read 0x019678 thru 0x026472 (0xcdfb bytes) 000546 1786752977.134895 mapread 0x0020f2 thru 0x00bb27 (0x9a36 bytes) 000548 1786752977.137352 mapwrite 0x005ed8 thru 0x00cb67 (0x6c90 bytes) 000551 1786752977.185409 trunc from 0x027574 to 0x020b0e (0x6a65 bytes) 000552 1786752977.196940 write 0x03f288 thru 0x03ffff (0xd78 bytes) 000553 1786752977.198081 write 0x032829 thru 0x034cc1 (0x2499 bytes) 000554 1786752977.205572 mapread 0x02086b thru 0x027702 (0x6e98 bytes) 000555 1786752977.207789 trunc from 0x040000 to 0x01812f (0x27ed0 bytes) 000556 1786752977.220992 trunc from 0x018130 to 0x00de81 (0xa2ae bytes) 000557 1786752977.237084 mapwrite 0x022060 thru 0x0319a9 (0xf94a bytes) 000558 1786752977.343360 read 0x02c5d2 thru 0x0319a9 (0x53d8 bytes) 000559 1786752977.344602 write 0x02f1dd thru 0x03d491 (0xe2b5 bytes) 000560 1786752977.346796 trunc from 0x03d492 to 0x03dad5 (0x644 bytes) 000561 1786752977.371277 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3dad6): Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dad6): Mirror 10: 0xc514baef (stale) Mirror 11: 0xc514baef (stale) CRC-32 checksum value for chunk [0, 0x3dad6): Mirror 1: 0x7c1a415c Mirror 12: 0x7c1a415c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dad6: skipped 000562 1786752978.095772 trunc from 0x03dad6 to 0x03f937 (0x1e62 bytes) 000563 1786752978.181008 write 0x039edf thru 0x03bc30 (0x1d52 bytes) 000564 1786752978.196452 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3f938): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f938): Mirror 10: 0xb10051b0 (stale) Mirror 11: 0xb10051b0 (stale) Mirror 12: 0x2c6b395d (stale) CRC-32 checksum value for chunk [0, 0x3f938): Mirror 1: 0x13eddee Mirror 13: 0x13eddee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f938: skipped 000565 1786752979.876821 trunc from 0x03f938 to 0x001597 (0x3e3a0 bytes) 000567 1786752979.986730 write 0x023140 thru 0x02d51b (0xa3dc bytes) 000568 1786752979.999910 MIRROR_RESYNC 5 mirrors 000569 1786752981.748358 mapread 0x00dc73 thru 0x016ccc (0x905a bytes) 000571 1786752981.803028 read 0x004045 thru 0x00b5c2 (0x757e bytes) 000573 1786752981.821566 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) 000575 1786752982.532526 mapwrite 0x014ffd thru 0x0164a2 (0x14a6 bytes) 000577 1786752982.790639 read 0x0095f3 thru 0x0110ec (0x7afa bytes) 000579 1786752982.805851 write 0x0195a6 thru 0x01e58c (0x4fe7 bytes) 000580 1786752982.830370 read 0x007fcf thru 0x00dc58 (0x5c8a bytes) 000581 1786752982.843231 read 0x029290 thru 0x02d51b (0x428c bytes) 000584 1786752982.845125 read 0x02502e thru 0x02d51b (0x84ee bytes) 000585 1786752982.850119 mapwrite 0x0237b2 thru 0x026c25 (0x3474 bytes) 000586 1786752982.962339 write 0x0243cb thru 0x02f41c (0xb052 bytes) 000587 1786752982.963792 write 0x0088eb thru 0x00d888 (0x4f9e bytes) 000589 1786752982.966706 write 0x01d179 thru 0x02085e (0x36e6 bytes) 000590 1786752982.969116 read 0x02e358 thru 0x02f41c (0x10c5 bytes) 000591 1786752982.970201 read 0x0253c8 thru 0x02f41c (0xa055 bytes) 000592 1786752982.970985 mapread 0x010429 thru 0x01f1c3 (0xed9b bytes) 000593 1786752982.977151 write 0x027f82 thru 0x02e13d (0x61bc bytes) 000596 1786752982.985605 mapread 0x024e97 thru 0x02ec43 (0x9dad bytes) 000597 1786752982.991714 mapwrite 0x00839a thru 0x016a98 (0xe6ff bytes) 000599 1786752983.069682 read 0x025b6e thru 0x02f41c (0x98af bytes) 000600 1786752983.071048 mapwrite 0x027795 thru 0x033776 (0xbfe2 bytes) 000601 1786752983.132045 trunc from 0x033777 to 0x0012e7 (0x3248f bytes) 000603 1786752983.199062 write 0x02277f thru 0x02faa2 (0xd324 bytes) 000606 1786752983.222156 mapwrite 0x03239d thru 0x03fe68 (0xdacc bytes) 000607 1786752983.293632 trunc from 0x03fe69 to 0x012e6d (0x2cffb bytes) 000608 1786752983.309515 write 0x00ccea thru 0x00fe70 (0x3187 bytes) 000611 1786752983.324104 read 0x011f6c thru 0x012e6d (0xf02 bytes) 000613 1786752983.340192 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12e6e): Mirror 11: 0xa754f902 (stale) Mirror 12: 0xa754f902 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12e6e): Mirror 11: 0xa754f902 (stale) Mirror 12: 0xa754f902 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12e6e: skipped 000615 1786752983.601176 mapwrite 0x0298ec thru 0x0327ac (0x8ec1 bytes) 000620 1786752983.684511 write 0x00b4cd thru 0x012b88 (0x76bc bytes) 000621 1786752983.694029 mapwrite 0x01b0dd thru 0x023a69 (0x898d bytes) 000624 1786752983.752967 write 0x026e3f thru 0x02cbb2 (0x5d74 bytes) 000625 1786752983.754537 trunc from 0x0327ad to 0x02876e (0xa03e bytes) 000628 1786752983.771704 read 0x00710a thru 0x0160fd (0xeff4 bytes) 000629 1786752983.780654 trunc from 0x02876f to 0x01663b (0x12133 bytes) 000630 1786752983.800874 mapwrite 0x011d92 thru 0x017eb2 (0x6121 bytes) 000631 1786752983.875300 trunc from 0x017eb3 to 0x01a65b (0x27a9 bytes) 000633 1786752983.897673 MIRROR_RESYNC 3 mirrors 000634 1786752984.837803 trunc from 0x01a65c to 0x00ed9f (0xb8bc bytes) 000636 1786752984.889893 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xeda0): Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xeda0): Mirror 11: 0xc6b8241 (stale) Mirror 12: 0xc6b8241 (stale) CRC-32 checksum value for chunk [0, 0xeda0): Mirror 1: 0xc6b8241 Mirror 13: 0xc6b8241 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xeda0: skipped 000637 1786752985.518848 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xeda0): Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xeda0): Mirror 11: 0xc6b8241 (stale) Mirror 12: 0xc6b8241 (stale) CRC-32 checksum value for chunk [0, 0xeda0): Mirror 1: 0xc6b8241 Mirror 13: 0xc6b8241 Mirror 14: 0xc6b8241 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xeda0: skipped 000638 1786752986.874325 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xeda0): Mirror 11: 0xc6b8241 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xeda0): Mirror 11: 0xc6b8241 (stale) CRC-32 checksum value for chunk [0, 0xeda0): Mirror 1: 0xc6b8241 Mirror 13: 0xc6b8241 Mirror 14: 0xc6b8241 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xeda0: skipped 000639 1786752988.012138 read 0x0051f0 thru 0x00c5dc (0x73ed bytes) 000640 1786752988.052868 read 0x001eac thru 0x008c74 (0x6dc9 bytes) 000642 1786752988.100133 read 0x000165 thru 0x00bfb5 (0xbe51 bytes) 000643 1786752988.112544 read 0x00a950 thru 0x00ed9f (0x4450 bytes) 000645 1786752988.170424 read 0x009c03 thru 0x00ed9f (0x519d bytes) 000648 1786752988.274944 trunc from 0x00eda0 to 0x01d6c0 (0xe921 bytes) 000649 1786752988.480896 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1d6c1): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d6c1): Mirror 11: 0x4054fdf8 (stale) Mirror 13: 0x3d7c444c (stale) Mirror 14: 0x3d7c444c (stale) CRC-32 checksum value for chunk [0, 0x1d6c1): Mirror 1: 0x3d7c444c Mirror 15: 0x3d7c444c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d6c1: skipped 000652 1786752989.834815 read 0x016da4 thru 0x01d6c0 (0x691d bytes) 000654 1786752989.862394 mapread 0x011cde thru 0x01ad1e (0x9041 bytes) 000655 1786752989.894564 trunc from 0x01d6c1 to 0x02a5a2 (0xcee2 bytes) 000656 1786752990.037997 write 0x00452d thru 0x00f1d8 (0xacac bytes) 000657 1786752990.051598 write 0x036bf6 thru 0x03ffff (0x940a bytes) 000658 1786752990.053215 mapread 0x016215 thru 0x01b2b0 (0x509c bytes) 000660 1786752990.061603 mapwrite 0x02bdcf thru 0x039641 (0xd873 bytes) 000662 1786752990.133496 mapread 0x02b2a5 thru 0x035edf (0xac3b bytes) 000664 1786752990.136913 read 0x025867 thru 0x02f5ae (0x9d48 bytes) 000665 1786752990.138185 mapread 0x02e459 thru 0x03c995 (0xe53d bytes) 000666 1786752990.140907 trunc from 0x040000 to 0x020fc0 (0x1f03f bytes) 000669 1786752990.169078 mapwrite 0x03df41 thru 0x03ffff (0x20bf bytes) 000673 1786752990.354807 mapwrite 0x00090e thru 0x000bbc (0x2af bytes) 000675 1786752990.466100 mapread 0x012e82 thru 0x02078d (0xd90c bytes) 000676 1786752990.470324 mapread 0x007056 thru 0x0104be (0x9469 bytes) 000677 1786752990.479849 read 0x001d1c thru 0x00bdba (0xa09f bytes) 000678 1786752990.483227 write 0x03c007 thru 0x03ffff (0x3ff9 bytes) 000680 1786752990.486065 mapwrite 0x029504 thru 0x029b3b (0x638 bytes) 000681 1786752990.623754 read 0x03262d thru 0x03acde (0x86b2 bytes) 000682 1786752990.624662 write 0x00f50d thru 0x01b6a0 (0xc194 bytes) 000684 1786752990.637956 read 0x01d380 thru 0x01e390 (0x1011 bytes) 000686 1786752990.639259 read 0x033433 thru 0x035a3d (0x260b bytes) 000688 1786752990.650169 write 0x03d3ed thru 0x03f2a2 (0x1eb6 bytes) 000689 1786752990.651225 mapread 0x03272c thru 0x039236 (0x6b0b bytes) 000692 1786752990.668496 write 0x0002fa thru 0x004241 (0x3f48 bytes) 000693 1786752990.676432 trunc from 0x040000 to 0x01fac7 (0x20538 bytes) 000694 1786752990.717080 trunc from 0x01fac8 to 0x017d97 (0x7d30 bytes) 000695 1786752990.773528 read 0x016e97 thru 0x017d97 (0xf01 bytes) 000697 1786752990.774760 mapwrite 0x03b219 thru 0x03ffff (0x4de7 bytes) 000698 1786752990.955106 mapread 0x015355 thru 0x0224f1 (0xd19d bytes) 000700 1786752990.960817 mapread 0x028b7c thru 0x029e8a (0x130f bytes) 000701 1786752990.971916 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xdde2e54e (stale) Mirror 13: 0x8b4ced50 (stale) Mirror 14: 0x8b4ced50 (stale) Mirror 15: 0x8b4ced50 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf50b55b6 Mirror 16: 0xf50b55b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000702 1786752992.525050 mapwrite 0x03aec8 thru 0x03ffff (0x5138 bytes) 000703 1786752992.752796 write 0x00912b thru 0x009c7d (0xb53 bytes) 000706 1786752992.761033 read 0x011f2c thru 0x013659 (0x172e bytes) 000707 1786752992.767870 write 0x0335ba thru 0x03a565 (0x6fac bytes) 000708 1786752992.773739 write 0x01efe5 thru 0x02d86a (0xe886 bytes) 000709 1786752992.785387 write 0x01cf64 thru 0x025dd0 (0x8e6d bytes) 000710 1786752992.793826 mapwrite 0x02dbb3 thru 0x0306f9 (0x2b47 bytes) 000712 1786752992.870180 mapwrite 0x01c17d thru 0x01d33b (0x11bf bytes) 000713 1786752992.918132 trunc from 0x040000 to 0x00bbb6 (0x34449 bytes) 000714 1786752992.942537 mapwrite 0x02c801 thru 0x0355ea (0x8dea bytes) 000716 1786752993.038772 trunc from 0x0355eb to 0x03827c (0x2c92 bytes) 000718 1786752993.108956 read 0x025db5 thru 0x02eb4f (0x8d9b bytes) 000719 1786752993.109759 write 0x0282a8 thru 0x028c2c (0x985 bytes) 000720 1786752993.116812 trunc from 0x03827d to 0x03545a (0x2e22 bytes) 000721 1786752993.170939 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3545b): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3545b): Mirror 11: 0x65498653 (stale) Mirror 13: 0x39476d14 (stale) Mirror 14: 0x39476d14 (stale) Mirror 15: 0x39476d14 (stale) Mirror 16: 0x1cf990ab (stale) CRC-32 checksum value for chunk [0, 0x3545b): Mirror 1: 0x6410127 Mirror 17: 0x6410127 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3545b: skipped 000725 1786752994.664656 mapwrite 0x0157d9 thru 0x018c47 (0x346f bytes) 000726 1786752995.028253 trunc from 0x03545b to 0x00d21c (0x2823e bytes) 000728 1786752995.091388 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd21d): Mirror 13: 0x69dc70b4 (stale) Mirror 14: 0x69dc70b4 (stale) Mirror 15: 0x69dc70b4 (stale) Mirror 16: 0xe56ef2a7 (stale) Mirror 17: 0x24522193 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd21d): Mirror 13: 0x69dc70b4 (stale) Mirror 14: 0x69dc70b4 (stale) Mirror 15: 0x69dc70b4 (stale) Mirror 16: 0xe56ef2a7 (stale) Mirror 17: 0x24522193 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd21d: skipped 000731 1786752995.524444 mapread 0x003a81 thru 0x00d21c (0x979c bytes) 000733 1786752995.541143 trunc from 0x00d21d to 0x01034e (0x3132 bytes) 000734 1786752995.556071 write 0x01c4fe thru 0x02671e (0xa221 bytes) 000738 1786752995.561442 mapread 0x010a2b thru 0x0161d0 (0x57a6 bytes) 000741 1786752995.571354 read 0x02088b thru 0x02671e (0x5e94 bytes) 000742 1786752995.572655 write 0x024241 thru 0x033d2f (0xfaef bytes) 000744 1786752995.575479 read 0x02f7b1 thru 0x033d2f (0x457f bytes) 000745 1786752995.576411 write 0x02b02b thru 0x03aa65 (0xfa3b bytes) 000746 1786752995.578802 write 0x0276b2 thru 0x02b281 (0x3bd0 bytes) 000747 1786752995.584222 MIRROR_RESYNC 6 mirrors 000749 1786752997.520447 read 0x016ea1 thru 0x018c28 (0x1d88 bytes) 000751 1786752997.529308 MIRROR_EXTEND to 7 mirrors 000752 1786752998.285909 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) 000754 1786752998.650347 MIRROR_RESYNC 6 mirrors 000755 1786752999.088185 read 0x010e9f thru 0x0175a4 (0x6706 bytes) 000756 1786752999.109917 read 0x0390ee thru 0x03aa65 (0x1978 bytes) 000759 1786752999.122198 mapread 0x03785a thru 0x03aa65 (0x320c bytes) 000760 1786752999.122860 trunc from 0x03aa66 to 0x01d80a (0x1d25b bytes) 000762 1786752999.236246 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d80b): Mirror 13: 0xbe5f321d (stale) Mirror 14: 0xbe5f321d (stale) Mirror 16: 0xbe5f321d (stale) Mirror 17: 0xbe5f321d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d80b): Mirror 13: 0xbe5f321d (stale) Mirror 14: 0xbe5f321d (stale) Mirror 16: 0xbe5f321d (stale) Mirror 17: 0xbe5f321d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d80b: skipped 000763 1786752999.666343 mapwrite 0x002180 thru 0x00b491 (0x9312 bytes) 000764 1786752999.743229 mapwrite 0x02be24 thru 0x038115 (0xc2f2 bytes) 000765 1786752999.943476 read 0x01ee1b thru 0x02209f (0x3285 bytes) 000767 1786752999.977126 mapread 0x01cc4e thru 0x02b195 (0xe548 bytes) 000770 1786753000.007266 mapwrite 0x032d7a thru 0x03ad6b (0x7ff2 bytes) 000773 1786753000.222045 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3ad6c): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ad6c): Mirror 13: 0xd8c50cf6 (stale) Mirror 14: 0xd8c50cf6 (stale) Mirror 16: 0xd8c50cf6 (stale) Mirror 17: 0xd8c50cf6 (stale) CRC-32 checksum value for chunk [0, 0x3ad6c): Mirror 1: 0x465f38e1 Mirror 18: 0x465f38e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ad6c: skipped 000774 1786753001.440721 mapread 0x00b31e thru 0x016d83 (0xba66 bytes) 000776 1786753001.546321 mapread 0x001ea3 thru 0x00867d (0x67db bytes) 000777 1786753001.599821 read 0x03753f thru 0x03ad6b (0x382d bytes) 000779 1786753001.636837 mapread 0x035df7 thru 0x039e6a (0x4074 bytes) 000780 1786753001.671718 read 0x0161e1 thru 0x01b368 (0x5188 bytes) 000781 1786753001.683560 mapwrite 0x013867 thru 0x0174c5 (0x3c5f bytes) 000783 1786753001.864937 trunc from 0x03ad6c to 0x02c58b (0xe7e0 bytes) 000785 1786753001.917898 write 0x03ebaf thru 0x03ffff (0x1451 bytes) 000786 1786753001.921228 write 0x02341d thru 0x025f51 (0x2b35 bytes) 000787 1786753001.928266 mapwrite 0x021187 thru 0x0297c8 (0x8642 bytes) 000791 1786753002.057015 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x483389f5 (stale) Mirror 14: 0x483389f5 (stale) Mirror 16: 0x483389f5 (stale) Mirror 17: 0x483389f5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x483389f5 (stale) Mirror 14: 0x483389f5 (stale) Mirror 16: 0x483389f5 (stale) Mirror 17: 0x483389f5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000794 1786753002.460254 mapwrite 0x01ac3f thru 0x01bb29 (0xeeb bytes) 000795 1786753002.545853 mapwrite 0x00bd5c thru 0x0171fa (0xb49f bytes) 000796 1786753002.643874 write 0x02784e thru 0x029073 (0x1826 bytes) 000798 1786753002.659156 write 0x012127 thru 0x01f379 (0xd253 bytes) 000799 1786753002.666478 read 0x002438 thru 0x007dff (0x59c8 bytes) 000800 1786753002.681555 mapwrite 0x0192d7 thru 0x01c68b (0x33b5 bytes) 000801 1786753002.715996 write 0x00f6ae thru 0x013ae5 (0x4438 bytes) 000802 1786753002.717199 read 0x036a3c thru 0x037897 (0xe5c bytes) 000804 1786753002.724873 write 0x00bcb8 thru 0x00c331 (0x67a bytes) 000806 1786753002.726724 mapwrite 0x01b7a4 thru 0x026945 (0xb1a2 bytes) 000807 1786753002.762925 write 0x01ba42 thru 0x024e1b (0x93da bytes) 000808 1786753002.764079 read 0x02849d thru 0x030432 (0x7f96 bytes) 000810 1786753002.765789 trunc from 0x040000 to 0x02a405 (0x15bfa bytes) 000811 1786753002.778009 mapwrite 0x006d43 thru 0x00e111 (0x73cf bytes) 000812 1786753002.862072 trunc from 0x02a406 to 0x02d8bc (0x34b7 bytes) 000816 1786753002.894318 mapwrite 0x03d626 thru 0x03ffff (0x29da bytes) 000817 1786753002.940355 mapwrite 0x025974 thru 0x0266ae (0xd3b bytes) 000819 1786753002.967649 write 0x0363e7 thru 0x0388f6 (0x2510 bytes) 000821 1786753002.969257 mapwrite 0x035d31 thru 0x03ffff (0xa2cf bytes) 000822 1786753003.008254 mapwrite 0x016d01 thru 0x020369 (0x9669 bytes) 000823 1786753003.062659 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x483389f5 (stale) Mirror 14: 0x483389f5 (stale) Mirror 17: 0x483389f5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x483389f5 (stale) Mirror 14: 0x483389f5 (stale) Mirror 17: 0x483389f5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000824 1786753003.369162 mapread 0x0353da thru 0x03df87 (0x8bae bytes) 000825 1786753003.382804 mapread 0x016e85 thru 0x026d51 (0xfecd bytes) 000829 1786753003.410736 mapwrite 0x0176f4 thru 0x0183fc (0xd09 bytes) 000831 1786753003.458918 mapread 0x015af7 thru 0x022afc (0xd006 bytes) 000832 1786753003.460362 trunc from 0x040000 to 0x03f8be (0x741 bytes) 000833 1786753003.477218 trunc from 0x03f8bf to 0x036fa1 (0x891d bytes) 000835 1786753003.498866 mapwrite 0x036436 thru 0x0379df (0x15aa bytes) 000836 1786753003.578353 mapread 0x021600 thru 0x030037 (0xea38 bytes) 000840 1786753003.589369 read 0x009e6e thru 0x00dba8 (0x3d3b bytes) 000843 1786753003.595229 write 0x0335c3 thru 0x039548 (0x5f86 bytes) 000845 1786753003.596560 mapread 0x034a9b thru 0x034fe4 (0x54a bytes) 000846 1786753003.597221 read 0x010239 thru 0x012ee3 (0x2cab bytes) 000849 1786753003.598789 read 0x02d427 thru 0x02eeca (0x1aa4 bytes) 000850 1786753003.599290 read 0x00e501 thru 0x0161f4 (0x7cf4 bytes) 000854 1786753003.606213 mapwrite 0x029dfd thru 0x0391d7 (0xf3db bytes) 000855 1786753003.648364 mapread 0x034573 thru 0x0357e3 (0x1271 bytes) 000856 1786753003.650241 mapread 0x01d4ba thru 0x0214f5 (0x403c bytes) 000857 1786753003.652067 mapread 0x01fbb8 thru 0x025d9b (0x61e4 bytes) 000858 1786753003.654330 write 0x009aa9 thru 0x018639 (0xeb91 bytes) 000860 1786753003.660195 mapread 0x03663a thru 0x039548 (0x2f0f bytes) 000861 1786753003.663968 trunc from 0x039549 to 0x02797b (0x11bcd bytes) 000862 1786753003.683582 read 0x0192ab thru 0x0193a6 (0xfc bytes) 000865 1786753003.686684 mapwrite 0x024e33 thru 0x025cc4 (0xe92 bytes) 000868 1786753003.744984 write 0x001b5b thru 0x00f481 (0xd927 bytes) 000870 1786753003.747710 mapwrite 0x01804f thru 0x0185c5 (0x577 bytes) 000871 1786753003.816955 write 0x00544f thru 0x012c73 (0xd825 bytes) 000872 1786753003.819917 mapwrite 0x019f28 thru 0x01c426 (0x24ff bytes) 000874 1786753003.869004 write 0x01d5cc thru 0x01ebd0 (0x1605 bytes) 000875 1786753003.870282 write 0x003ab9 thru 0x00b12e (0x7676 bytes) 000876 1786753003.871739 read 0x0002ac thru 0x00dd4e (0xdaa3 bytes) 000877 1786753003.876023 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2797c): Mirror 14: 0x1c806887 (stale) Mirror 17: 0x1c806887 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2797c): Mirror 14: 0x1c806887 (stale) Mirror 17: 0x1c806887 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2797c: skipped 000881 1786753004.142240 read 0x00e5a8 thru 0x0149b6 (0x640f bytes) 000884 1786753004.158666 read 0x01c9c0 thru 0x02797b (0xafbc bytes) 000885 1786753004.199163 mapwrite 0x0192c8 thru 0x024177 (0xaeb0 bytes) 000891 1786753004.233664 write 0x006396 thru 0x00bda3 (0x5a0e bytes) 000892 1786753004.234911 mapwrite 0x0062b2 thru 0x00e5ca (0x8319 bytes) 000893 1786753004.262809 read 0x021c48 thru 0x02797b (0x5d34 bytes) 000894 1786753004.266196 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2797c): Mirror 17: 0x1c806887 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2797c): Mirror 17: 0x1c806887 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2797c: skipped 000895 1786753004.466067 trunc from 0x02797c to 0x020e8f (0x6aec bytes) 000897 1786753004.488732 trunc from 0x020e90 to 0x022dc7 (0x1f38 bytes) 000898 1786753004.506893 read 0x0143f7 thru 0x015ebd (0x1ac7 bytes) 000900 1786753004.514001 mapwrite 0x03ef55 thru 0x03ffff (0x10ab bytes) 000904 1786753004.672099 write 0x00859a thru 0x0183c6 (0xfe2d bytes) 000905 1786753004.685347 mapwrite 0x007cdf thru 0x00cb73 (0x4e95 bytes) 000906 1786753004.766763 write 0x00184b thru 0x00db33 (0xc2e9 bytes) 000907 1786753004.769854 mapwrite 0x036b95 thru 0x03a9f5 (0x3e61 bytes) 000910 1786753004.864911 MIRROR_RESYNC 2 mirrors 000911 1786753005.723223 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) 000917 1786753005.894689 trunc from 0x040000 to 0x013d98 (0x2c267 bytes) 000918 1786753005.922007 mapread 0x01189b thru 0x012376 (0xadc bytes) 000919 1786753005.932724 read 0x011c98 thru 0x011e43 (0x1ac bytes) 000925 1786753005.939744 mapwrite 0x0236cb thru 0x031484 (0xddba bytes) 000926 1786753006.012137 mapwrite 0x02183a thru 0x027772 (0x5f39 bytes) 000927 1786753006.055681 read 0x007a69 thru 0x00a971 (0x2f09 bytes) 000928 1786753006.065067 mapread 0x02b82f thru 0x031484 (0x5c56 bytes) 000931 1786753006.068835 trunc from 0x031485 to 0x00129c (0x301e8 bytes) 000936 1786753006.086911 mapwrite 0x0100e1 thru 0x0112e8 (0x1208 bytes) 000938 1786753006.163227 write 0x00d210 thru 0x01b86b (0xe65c bytes) 000941 1786753006.177305 write 0x00d4f8 thru 0x01893c (0xb445 bytes) 000943 1786753006.188072 mapread 0x0080b2 thru 0x009b0c (0x1a5b bytes) 000944 1786753006.199422 mapread 0x0124db thru 0x016648 (0x416e bytes) 000947 1786753006.216985 MIRROR_EXTEND to 2 mirrors 000949 1786753007.622256 mapread 0x00aa26 thru 0x01a8e1 (0xfebc bytes) 000951 1786753007.738090 write 0x0340f9 thru 0x03fe62 (0xbd6a bytes) 000953 1786753007.913822 trunc from 0x03fe63 to 0x02ffea (0xfe78 bytes) 000954 1786753007.952164 trunc from 0x02ffeb to 0x01b62a (0x149c0 bytes) 000955 1786753007.986780 trunc from 0x01b62b to 0x01f5b5 (0x3f8b bytes) 000957 1786753008.040175 trunc from 0x01f5b6 to 0x00c160 (0x13455 bytes) 000958 1786753008.081691 write 0x019ef5 thru 0x01a912 (0xa1e bytes) 000959 1786753008.103353 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) 000960 1786753008.366587 mapwrite 0x02de5c thru 0x036874 (0x8a19 bytes) 000961 1786753008.552823 trunc from 0x036875 to 0x00d0d9 (0x2979b bytes) 000962 1786753008.579835 read 0x00d04a thru 0x00d0d9 (0x90 bytes) 000963 1786753008.586075 write 0x03cf78 thru 0x03ffff (0x3088 bytes) 000964 1786753008.592680 MIRROR_EXTEND to 2 mirrors 000966 1786753009.312901 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) 000967 1786753009.394766 trunc from 0x040000 to 0x004dca (0x3b235 bytes) 000969 1786753009.413083 read 0x004169 thru 0x004dca (0xc62 bytes) 000970 1786753009.417889 read 0x0019a2 thru 0x004dca (0x3429 bytes) 000971 1786753009.423998 mapread 0x004366 thru 0x004dca (0xa65 bytes) 000974 1786753009.426188 mapwrite 0x014c20 thru 0x01d525 (0x8906 bytes) 000976 1786753009.499216 mapread 0x0078ff thru 0x0106ef (0x8df1 bytes) 000977 1786753009.501734 read 0x0149fb thru 0x01c62b (0x7c31 bytes) 000978 1786753009.502626 mapread 0x019f41 thru 0x01d525 (0x35e5 bytes) 000979 1786753009.504599 write 0x01c470 thru 0x020cd7 (0x4868 bytes) 000980 1786753009.506806 trunc from 0x020cd8 to 0x02710b (0x6434 bytes) 000981 1786753009.572189 read 0x01fbda thru 0x0221a1 (0x25c8 bytes) 000984 1786753009.594786 write 0x0305c6 thru 0x038b74 (0x85af bytes) 000985 1786753009.620175 MIRROR_EXTEND to 2 mirrors 000986 1786753011.657873 trunc from 0x038b75 to 0x0336d0 (0x54a4 bytes) 000987 1786753011.780036 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) 000989 1786753012.043410 trunc from 0x0336d1 to 0x03da35 (0xa365 bytes) 000991 1786753012.069713 mapread 0x009c38 thru 0x018ef4 (0xf2bd bytes) 000994 1786753012.108446 mapwrite 0x0139c2 thru 0x013a7f (0xbe bytes) 000995 1786753012.173790 read 0x03aff5 thru 0x03da35 (0x2a41 bytes) 001000 1786753012.188617 read 0x01eb31 thru 0x02d870 (0xed40 bytes) All operations completed A-OK! PASS 70a (107s) == sanity-flr test 71: check mirror extend parent fid ==== 20:16:58 (1786753018) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 20:17:02 (1786753022) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.306496 s, 6.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201839 s, 5.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ba:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ba:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d7:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 20:17:12 (1786753032) Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3009728 .. Extending file size to 5453120 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9371552 .. Extending file size to 5553728 .. Extending file size to 7699680 .. Extending file size to 8393760 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8522816 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9163456 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9321280 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 57396 57397 57400 57401 57402 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) 192.168.201.111@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (78s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:18:30 (1786753110) Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6388864 .. Extending file size to 5424160 .. Extending file size to 7407520 .. Extending file size to 8931008 .. Extending file size to 8459712 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8674624 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8733280 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9223456 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58910 58911 58912 58914 58915 done 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) 192.168.201.111@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (76s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:19:46 (1786753186) Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.106674 s, 19.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.269228 s, 7.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x45f75388 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:20:04 (1786753204) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0793752 s, 26.4 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 20:20:10 (1786753210) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e6:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:20:17 (1786753217) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.161894 s, 13.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (6s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:20:23 (1786753223) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.12946 s, 16.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f0:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:20:31 (1786753231) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.18648 s, 11.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:21:04 (1786753264) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg111-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123006 s, 8.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:21:59 (1786753319) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg111-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.406847 s, 5.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:22:57 (1786753377) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg111-server mds-ost sync done. Creating new pool test_204f oleg111-server: Pool lustre.test_204f created Adding targets to pool oleg111-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg111-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.276888 s, 7.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg111-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg111-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg111-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:24:17 (1786753457) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:24:25 (1786753465) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:24:33 (1786753473) oleg111-server: Pool lustre.test_206 created Adding targets to pool oleg111-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg111-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg111-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg111-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg111-server: Pool lustre.test_206 destroyed PASS 206 (23s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:24:56 (1786753496) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523392 s, 20.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377296 s, 27.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x610e7ca4 Mirror 3: 0x610e7ca4 PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:25:05 (1786753505) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:25:09 (1786753509) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:25:12 (1786753512) Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.106039 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1ac0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x730000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x730000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x730000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1c40000 Read this (4 bytes): BBBB 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:25:51 (1786753551) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00923358 s, 887 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:25:58 (1786753558) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476432 s, 22.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20b lmm_fid: [0x200000404:0x20b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20b lmm_fid: [0x200000404:0x20b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834443 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-14 20:25:59.000000000 -0400 Modify: 2026-08-14 20:25:59.000000000 -0400 Change: 2026-08-14 20:26:00.000000000 -0400 Birth: 2026-08-14 20:25:59.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:26:06 (1786753566) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:26:09 (1786753569) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.640735 s, 63.9 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.03943 s, 104 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 14 20:26 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20d:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 14 20:26 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:26:17 (1786753577) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:26:25 (1786753585) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1765 sec =========== 20:26:35 (1786753595) === sanity-flr: start cleanup 20:26:36 (1786753596) === === sanity-flr: finish cleanup 20:26:45 (1786753605) ===