-----============= acceptance-small: sanity-flr ============----- Sat Aug 15 16:44:54 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg326-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 16:45:12 (1786826712) === oleg326-client.virtnet: executing check_config_client /mnt/lustre oleg326-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg326-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff98aa42cc0000.idle_timeout=debug osc.lustre-OST0001-osc-ffff98aa42cc0000.idle_timeout=debug disable quota as required oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 16:45:42 (1786826742) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7616] == sanity-flr test 0a: lfs mirror create with -N option == 16:45:44 (1786826744) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 16:46:01 (1786826761) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 16:46:04 (1786826764) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 16:46:07 (1786826767) PASS 0d (21s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 16:46:28 (1786826788) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 16:46:31 (1786826791) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 16:46:35 (1786826795) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff98aa42cc0000.stats=clear osc.lustre-OST0001-osc-ffff98aa42cc0000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 16:46:42 (1786826802) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 16:46:52 (1786826812) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 16:47:01 (1786826821) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 16:47:03 (1786826823) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 16:47:10 (1786826830) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 16:47:19 (1786826839) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 16:47:26 (1786826846) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 16:47:29 (1786826849) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (5s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 16:47:34 (1786826854) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (5s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 16:47:42 (1786826862) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 16:47:48 (1786826868) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 1.53301 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680315 s, 15.4 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 16:47:57 (1786826877) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.106208 s, 29.6 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 16:48:04 (1786826884) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (8s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 16:48:12 (1786826892) Stopping /mnt/lustre-ost1 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98aa42cc0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98aa42cc0000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg326-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98aa42cc0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98aa42cc0000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg326-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (56s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 16:49:08 (1786826948) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.276632 s, 30.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98aa42cc0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98aa42cc0000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg326-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (31s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 16:49:40 (1786826980) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98aa42cc0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98aa42cc0000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg326-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98aa42cc0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98aa42cc0000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg326-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (58s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 16:50:37 (1786827037) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.55589 s, 14.7 MB/s -rw-r--r-- 1 root root 52428800 Aug 15 16:50 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.720s user 0m0.001s sys 0m0.182s Stopping /mnt/lustre-ost1 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98aa42cc0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98aa42cc0000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.838s user 0m0.001s sys 0m0.189s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg326-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98aa42cc0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98aa42cc0000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.957s user 0m0.001s sys 0m0.162s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg326-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg326-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98aa42cc0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98aa42cc0000.ost_server_uuid in FULL state after 0 sec PASS 33b (68s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 16:51:45 (1786827105) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 16:51:47 (1786827107) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 16:51:50 (1786827110) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 16:51:53 (1786827113) PASS 35 (5s) == sanity-flr test 36a: write to mirrored files ========== 16:51:58 (1786827118) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.395204 s, 31.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (10s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 16:52:08 (1786827128) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270056 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327552 s, 32.0 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 16:52:14 (1786827134) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 16:52:22 (1786827142) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00125706 s, 815 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000939346 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00101864 s, 1.0 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00207458 s, 494 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 16:52:31 (1786827151) create test files with size 15751054 .. c0a826e750af0ec54a341ab812d11e65 - c00dceee5265c5257d24052c4b4d11fd - 0b95ce5e72280f00d66361e3f4879755 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg326-client.virtnet /mnt/lustre (opts:) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre PASS 37 (19s) == sanity-flr test 38: resync ============================ 16:52:50 (1786827170) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 16:53:04 (1786827184) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 16:53:12 (1786827192) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223752 s, 4.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc4:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303702 s, 34.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc5:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc6:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd9:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (12s) == sanity-flr test 41: lfs mirror resync check =========== 16:53:23 (1786827203) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.117348 s, 17.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.398706 s, 10.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.22076 s, 9.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 16:53:38 (1786827218) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 16:53:41 (1786827221) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 16:53:43 (1786827223) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330549 s, 31.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 16:53:49 (1786827229) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 16:53:52 (1786827232) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcd:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe0:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe0:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (6s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 16:53:59 (1786827239) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.249102 s, 42.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg326-client.virtnet /mnt/lustre (opts:) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 16:54:09 (1786827249) llite.lustre-ffff98aa434d4800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.361483 s, 29.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg326-client.virtnet /mnt/lustre (opts:) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff98aa58dc1800.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 16:54:22 (1786827262) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (30s) == sanity-flr test 46: Verify setstripe --copy option ==== 16:54:52 (1786827292) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (28s) == sanity-flr test 47: Verify mirror obj alloc =========== 16:55:20 (1786827320) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 16:55:23 (1786827323) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.259552 s, 12.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.180909 s, 17.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xfe605a10 (nosync,stale) Mirror 2: 0xfe605a10 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 2ce4c9781a393856f6b11b7babc24d76 - mirror 1 checksum: 49ed398baed0eeccaf7d69a52b409681 - mirror 2 checksum: 2ce4c9781a393856f6b11b7babc24d76 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-15T16:55:25-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-15T16:55:25-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe4:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 2ce4c9781a393856f6b11b7babc24d76 - mirror 1 checksum: 49ed398baed0eeccaf7d69a52b409681 - mirror 2 checksum: 49ed398baed0eeccaf7d69a52b409681 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-15T16:55:25-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-15T16:55:25-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe4:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 16:55:34 (1786827334) Failing mds1 on oleg326-server Stopping /mnt/lustre-mds1 (opts:) on oleg326-server 16:55:41 (1786827341) shut down facet: mds1 facet_host: oleg326-server facet_failover_host: oleg326-server Failover mds1 to oleg326-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-MDT0000 16:56:02 (1786827362) targets are mounted 16:56:02 (1786827362) facet_failover done oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300555 s, 34.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe5:0x0] } PASS 50A (47s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 16:56:21 (1786827381) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 16:56:26 (1786827386) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 16:56:29 (1786827389) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00888107 s, 1.8 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 16:56:38 (1786827398) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 16:56:41 (1786827401) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0165729 s, 247 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.105326 s, 38.9 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 16:56:51 (1786827411) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 16:57:14 (1786827434) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 16:57:43 (1786827463) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 16:58:07 (1786827487) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 16:58:30 (1786827510) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 16:58:37 (1786827517) mirror create pid 54262 mirror split pid 54263 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54262 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54263 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 16:59:44 (1786827584) Seed set to 7586 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1786827587.357756 write 0x012f8c thru 0x017048 (0x40bd bytes) truncating to largest ever: 0x146e3 000005 1786827587.373600 trunc from 0x017049 to 0x0146e2 (0x2966 bytes) 000006 1786827587.414853 write 0x01cde8 thru 0x0288ce (0xbae7 bytes) 000007 1786827587.422520 mapread 0x01efec thru 0x027479 (0x848e bytes) 000008 1786827587.435365 mapwrite 0x012bf0 thru 0x022a09 (0xfe1a bytes) 000009 1786827587.532456 mapread 0x0249ae thru 0x0286d2 (0x3d25 bytes) 000010 1786827587.534989 mapread 0x01a57c thru 0x0256d8 (0xb15d bytes) 000013 1786827587.538381 mapwrite 0x017b00 thru 0x01d337 (0x5838 bytes) 000015 1786827587.568373 read 0x003b54 thru 0x004274 (0x721 bytes) 000016 1786827587.575505 mapread 0x00434f thru 0x0055e4 (0x1296 bytes) truncating to largest ever: 0x35454 000019 1786827587.588516 trunc from 0x0288cf to 0x035453 (0xcb85 bytes) 000020 1786827587.623994 MIRROR_EXTEND to 2 mirrors 000022 1786827588.500299 trunc from 0x035454 to 0x0179b8 (0x1da9b bytes) 000023 1786827588.650422 read 0x00af18 thru 0x015416 (0xa4ff bytes) 000024 1786827588.664758 mapread 0x0104f3 thru 0x0179b8 (0x74c6 bytes) 000025 1786827588.681114 mapread 0x010cbc thru 0x0118ea (0xc2f bytes) truncating to largest ever: 0x3b6e6 000026 1786827588.682410 trunc from 0x0179b9 to 0x03b6e5 (0x23d2d bytes) 000027 1786827588.700277 read 0x0182f9 thru 0x020c43 (0x894b bytes) 000028 1786827588.714410 trunc from 0x03b6e6 to 0x00df71 (0x2d774 bytes) 000029 1786827588.739432 MIRROR_RESYNC 2 mirrors 000031 1786827589.482411 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) 000032 1786827589.641824 mapwrite 0x02c7d6 thru 0x03a08e (0xd8b9 bytes) 000033 1786827589.724810 mapread 0x01c5dc thru 0x01d923 (0x1348 bytes) 000035 1786827589.738878 trunc from 0x03a08f to 0x02f236 (0xae58 bytes) 000036 1786827589.758052 trunc from 0x02f237 to 0x01c1fe (0x13038 bytes) 000037 1786827589.775428 read 0x005e05 thru 0x007c65 (0x1e61 bytes) 000038 1786827589.789300 mapread 0x01af63 thru 0x01c1fe (0x129c bytes) 000040 1786827589.794051 trunc from 0x01c1ff to 0x021d88 (0x5b8a bytes) 000041 1786827589.816346 mapread 0x01cf0c thru 0x021d88 (0x4e7d bytes) 000042 1786827589.827481 mapread 0x01e8bc thru 0x021b49 (0x328e bytes) 000043 1786827589.831987 write 0x038efb thru 0x03ffff (0x7105 bytes) 000049 1786827589.843409 write 0x02b3d4 thru 0x03287d (0x74aa bytes) 000051 1786827589.868184 mapwrite 0x008062 thru 0x0171bf (0xf15e bytes) 000052 1786827589.954978 mapread 0x02e9de thru 0x03ca05 (0xe028 bytes) 000057 1786827589.991324 mapread 0x02dcfa thru 0x03b1b4 (0xd4bb bytes) 000058 1786827589.994677 write 0x02cce9 thru 0x0325ce (0x58e6 bytes) 000059 1786827589.996184 write 0x003dfb thru 0x00e72d (0xa933 bytes) 000060 1786827590.009517 write 0x00c70b thru 0x00e066 (0x195c bytes) 000061 1786827590.013309 mapwrite 0x0254d2 thru 0x033e34 (0xe963 bytes) 000062 1786827590.104499 mapread 0x027360 thru 0x030574 (0x9215 bytes) 000063 1786827590.111364 read 0x007d80 thru 0x016e85 (0xf106 bytes) 000067 1786827590.116705 mapread 0x01f046 thru 0x020696 (0x1651 bytes) 000068 1786827590.117807 trunc from 0x040000 to 0x022a96 (0x1d569 bytes) 000073 1786827590.142293 mapwrite 0x036d2a thru 0x03cbdc (0x5eb3 bytes) 000081 1786827590.253441 mapwrite 0x02cef1 thru 0x036674 (0x9784 bytes) 000082 1786827590.312397 read 0x02cc1f thru 0x038525 (0xb907 bytes) 000083 1786827590.313039 trunc from 0x03cbdd to 0x00f0ce (0x2db0e bytes) 000084 1786827590.336825 write 0x02eb36 thru 0x0340c2 (0x558d bytes) 000088 1786827590.348042 mapread 0x0154bb thru 0x0254ae (0xfff4 bytes) 000089 1786827590.359935 write 0x038a9a thru 0x03ffff (0x7566 bytes) 000091 1786827590.362014 write 0x0266fd thru 0x02aee7 (0x47eb bytes) 000092 1786827590.368937 mapwrite 0x033ece thru 0x036462 (0x2595 bytes) 000093 1786827590.478337 trunc from 0x040000 to 0x002570 (0x3da8f bytes) 000094 1786827590.514058 mapwrite 0x03b130 thru 0x03ffff (0x4ed0 bytes) 000096 1786827590.688071 mapread 0x00ffd3 thru 0x01ec9f (0xeccd bytes) 000097 1786827590.800410 mapwrite 0x02f85c thru 0x036383 (0x6b28 bytes) 000099 1786827590.936038 mapwrite 0x031b35 thru 0x0340f1 (0x25bd bytes) 000100 1786827591.067010 mapwrite 0x0335f0 thru 0x03ffff (0xca10 bytes) 000101 1786827591.245367 read 0x02a117 thru 0x031892 (0x777c bytes) 000106 1786827591.276040 write 0x018ad3 thru 0x026a54 (0xdf82 bytes) 000107 1786827591.277806 trunc from 0x040000 to 0x01272c (0x2d8d3 bytes) 000108 1786827591.303722 write 0x037c16 thru 0x03ffff (0x83ea bytes) 000110 1786827591.305748 mapread 0x00ba87 thru 0x017ec0 (0xc43a bytes) truncating to largest ever: 0x3e42e 000112 1786827591.339875 trunc from 0x040000 to 0x03e42d (0x1bd2 bytes) 000114 1786827591.360659 mapwrite 0x0301e1 thru 0x0363c2 (0x61e2 bytes) 000115 1786827591.438360 write 0x03cef0 thru 0x03ffff (0x3110 bytes) 000116 1786827591.440001 mapwrite 0x011e2d thru 0x01cc55 (0xae29 bytes) 000117 1786827591.513472 mapwrite 0x0206cf thru 0x02933f (0x8c71 bytes) 000119 1786827591.650681 mapread 0x00b1a7 thru 0x012488 (0x72e2 bytes) 000120 1786827591.652994 mapread 0x03e023 thru 0x03ffff (0x1fdd bytes) 000122 1786827591.661429 read 0x035386 thru 0x03aba7 (0x5822 bytes) 000124 1786827591.664721 trunc from 0x040000 to 0x02d91f (0x126e0 bytes) 000125 1786827591.707161 mapwrite 0x01ffcd thru 0x02bae5 (0xbb19 bytes) 000126 1786827591.781260 mapread 0x00ba2d thru 0x0108f7 (0x4ecb bytes) 000132 1786827591.787008 mapread 0x006ef6 thru 0x00f420 (0x852b bytes) 000133 1786827591.788903 read 0x0245de thru 0x024a35 (0x458 bytes) 000134 1786827591.789857 write 0x00d649 thru 0x01b94f (0xe307 bytes) 000137 1786827591.793613 mapread 0x0098f1 thru 0x012bc6 (0x92d6 bytes) 000140 1786827591.797279 mapread 0x02aca3 thru 0x02d91f (0x2c7d bytes) 000141 1786827591.798818 read 0x029aef thru 0x02d91f (0x3e31 bytes) 000143 1786827591.800541 trunc from 0x02d920 to 0x0111d0 (0x1c74f bytes) 000146 1786827591.818835 write 0x005ed8 thru 0x011917 (0xba40 bytes) 000147 1786827591.821330 read 0x0071e9 thru 0x011917 (0xa72f bytes) 000148 1786827591.823304 write 0x00b697 thru 0x0113a9 (0x5d13 bytes) 000149 1786827591.825277 mapread 0x00f5bb thru 0x011917 (0x235d bytes) 000150 1786827591.827011 read 0x00eb3e thru 0x011917 (0x2dda bytes) 000151 1786827591.827999 mapread 0x001598 thru 0x003aa7 (0x2510 bytes) 000152 1786827591.829378 trunc from 0x011918 to 0x00dfe7 (0x3930 bytes) 000154 1786827591.845556 mapread 0x0003aa thru 0x00bab3 (0xb70a bytes) 000155 1786827591.848240 trunc from 0x00dfe8 to 0x00c803 (0x17e4 bytes) 000156 1786827591.887678 mapwrite 0x030558 thru 0x03c5ed (0xc096 bytes) 000158 1786827592.013564 read 0x036020 thru 0x03c5ed (0x65ce bytes) 000160 1786827592.017844 trunc from 0x03c5ee to 0x0347f9 (0x7df4 bytes) 000161 1786827592.061836 read 0x015326 thru 0x01bc03 (0x68de bytes) 000163 1786827592.070184 mapwrite 0x0195da thru 0x01d376 (0x3d9d bytes) 000164 1786827592.197637 trunc from 0x0347fa to 0x01321c (0x215dd bytes) 000165 1786827592.241285 read 0x00b6b3 thru 0x011459 (0x5da7 bytes) 000166 1786827592.245248 trunc from 0x01321d to 0x01ebcb (0xb9af bytes) 000167 1786827592.270260 trunc from 0x01ebcc to 0x03b680 (0x1cab5 bytes) 000168 1786827592.323271 MIRROR_EXTEND to 2 mirrors 000169 1786827593.261084 mapread 0x00dd13 thru 0x0105c3 (0x28b1 bytes) 000170 1786827593.291728 read 0x00812a thru 0x01336d (0xb244 bytes) 000172 1786827593.292717 trunc from 0x03b681 to 0x0047f5 (0x36e8b bytes) 000174 1786827593.367397 mapwrite 0x03d08b thru 0x03ffff (0x2f75 bytes) 000175 1786827593.520253 MIRROR_RESYNC 2 mirrors 000179 1786827594.308666 mapread 0x033d4c thru 0x0399f4 (0x5ca9 bytes) 000181 1786827594.324804 mapwrite 0x035ce2 thru 0x03ffff (0xa31e bytes) 000182 1786827594.458928 mapread 0x01d510 thru 0x022dfb (0x58ec bytes) 000183 1786827594.470622 trunc from 0x040000 to 0x022d0f (0x1d2f0 bytes) 000187 1786827594.509583 read 0x015abf thru 0x01df0e (0x8450 bytes) 000188 1786827594.516969 trunc from 0x022d10 to 0x00d761 (0x155ae bytes) 000189 1786827594.555015 write 0x023619 thru 0x026973 (0x335b bytes) 000191 1786827594.558304 write 0x012365 thru 0x01d972 (0xb60e bytes) 000193 1786827594.568681 write 0x03ca5e thru 0x03ffff (0x35a2 bytes) 000194 1786827594.570581 read 0x00d8a5 thru 0x00fe9c (0x25f8 bytes) 000196 1786827594.580765 mapread 0x03e4f4 thru 0x03ef9d (0xaaa bytes) 000200 1786827594.584508 mapread 0x026d65 thru 0x034a14 (0xdcb0 bytes) 000202 1786827594.588227 write 0x028d91 thru 0x02d476 (0x46e6 bytes) 000203 1786827594.589627 read 0x03666e thru 0x036a9a (0x42d bytes) 000204 1786827594.591337 trunc from 0x040000 to 0x03dd51 (0x22ae bytes) 000205 1786827594.605370 read 0x008c26 thru 0x0157ff (0xcbda bytes) 000206 1786827594.606517 mapread 0x020736 thru 0x023955 (0x3220 bytes) 000208 1786827594.609496 mapread 0x01fe38 thru 0x02bd0b (0xbed4 bytes) 000209 1786827594.611519 mapread 0x01481e thru 0x017a7c (0x325f bytes) 000211 1786827594.617792 trunc from 0x03dd52 to 0x00bc88 (0x320c9 bytes) 000212 1786827594.632317 mapread 0x00489b thru 0x00bc88 (0x73ee bytes) 000213 1786827594.634589 mapwrite 0x010d2f thru 0x01e297 (0xd569 bytes) 000217 1786827594.718194 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1e298): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e298): Mirror 2: 0xe8728e3d (stale) CRC-32 checksum value for chunk [0, 0x1e298): Mirror 1: 0x1b41debe Mirror 3: 0x1b41debe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e298: skipped 000219 1786827595.530997 mapread 0x01536b thru 0x018998 (0x362e bytes) 000220 1786827595.563662 mapread 0x01b659 thru 0x01e297 (0x2c3f bytes) 000223 1786827595.593541 trunc from 0x01e298 to 0x03b9c6 (0x1d72f bytes) 000224 1786827595.683021 read 0x0280dd thru 0x02c681 (0x45a5 bytes) 000225 1786827595.690482 read 0x020f06 thru 0x0235af (0x26aa bytes) 000227 1786827595.731583 mapwrite 0x02e9aa thru 0x039135 (0xa78c bytes) 000228 1786827595.796929 write 0x038276 thru 0x03c9f7 (0x4782 bytes) 000229 1786827595.799580 mapwrite 0x024dd1 thru 0x02d49a (0x86ca bytes) 000231 1786827595.866728 mapwrite 0x006c78 thru 0x014348 (0xd6d1 bytes) 000233 1786827595.922292 read 0x02e251 thru 0x033364 (0x5114 bytes) 000234 1786827595.923338 write 0x01a9d3 thru 0x01b235 (0x863 bytes) 000235 1786827595.925309 mapwrite 0x011651 thru 0x021036 (0xf9e6 bytes) 000236 1786827595.971128 read 0x0324ca thru 0x03761e (0x5155 bytes) 000237 1786827595.972394 mapwrite 0x021a9a thru 0x02cc69 (0xb1d0 bytes) 000238 1786827596.014961 mapwrite 0x001bdc thru 0x00fbe1 (0xe006 bytes) 000239 1786827596.066772 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3c9f8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c9f8): Mirror 2: 0x9e6e5d63 (stale) Mirror 3: 0x212aae2d (stale) CRC-32 checksum value for chunk [0, 0x3c9f8): Mirror 1: 0x5e55b78b Mirror 4: 0x5e55b78b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c9f8: skipped 000240 1786827596.966597 trunc from 0x03c9f8 to 0x01b978 (0x2107f bytes) 000242 1786827597.086796 trunc from 0x01b979 to 0x02d877 (0x11eff bytes) 000243 1786827597.116131 mapread 0x0217ff thru 0x02ab74 (0x9376 bytes) 000247 1786827597.144190 trunc from 0x02d878 to 0x020cd6 (0xcba1 bytes) 000249 1786827597.168974 trunc from 0x020cd7 to 0x00d95c (0x1337a bytes) 000252 1786827597.193259 mapread 0x005883 thru 0x00d95c (0x80da bytes) 000253 1786827597.217344 trunc from 0x00d95d to 0x02237a (0x14a1e bytes) 000254 1786827597.235156 mapread 0x00e0ff thru 0x01836b (0xa26d bytes) 000255 1786827597.255284 trunc from 0x02237b to 0x03c793 (0x1a419 bytes) 000258 1786827597.281003 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3c794): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c794): Mirror 2: 0x40156d87 (stale) Mirror 3: 0x60c47b1 (stale) Mirror 4: 0xb017d0cf (stale) CRC-32 checksum value for chunk [0, 0x3c794): Mirror 1: 0x9c344eee Mirror 5: 0x9c344eee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c794: skipped 000260 1786827598.298071 read 0x02ecc8 thru 0x030416 (0x174f bytes) 000261 1786827598.303757 read 0x009ac4 thru 0x01287d (0x8dba bytes) 000262 1786827598.314573 trunc from 0x03c794 to 0x03adb2 (0x19e1 bytes) 000264 1786827598.416698 read 0x01da95 thru 0x0280b6 (0xa622 bytes) 000265 1786827598.433853 MIRROR_RESYNC 5 mirrors 000266 1786827599.630846 mapread 0x030dac thru 0x032c4e (0x1ea3 bytes) 000270 1786827599.662588 write 0x02c626 thru 0x030fe2 (0x49bd bytes) 000271 1786827599.747603 mapwrite 0x0294dd thru 0x032335 (0x8e59 bytes) 000273 1786827599.830438 mapwrite 0x01479c thru 0x01a4f9 (0x5d5e bytes) 000274 1786827599.903053 read 0x0118b0 thru 0x014d47 (0x3498 bytes) 000276 1786827599.905813 trunc from 0x03adb3 to 0x01e789 (0x1c629 bytes) 000277 1786827599.935689 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1e78a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e78a): Mirror 2: 0x9403880a (stale) Mirror 3: 0x9403880a (stale) Mirror 4: 0x9403880a (stale) Mirror 5: 0x9403880a (stale) CRC-32 checksum value for chunk [0, 0x1e78a): Mirror 1: 0x91b742ab Mirror 6: 0x91b742ab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e78a: skipped 000278 1786827600.988720 trunc from 0x01e78a to 0x00e532 (0x10257 bytes) 000280 1786827601.092910 trunc from 0x00e533 to 0x027d7d (0x1984b bytes) 000281 1786827601.117154 mapread 0x022ff6 thru 0x027d7d (0x4d88 bytes) 000283 1786827601.171447 write 0x026ede thru 0x02c497 (0x55ba bytes) 000284 1786827601.173366 mapwrite 0x023fb0 thru 0x030cf5 (0xcd46 bytes) 000285 1786827601.271243 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30cf6): Mirror 2: 0x1294af1 (stale) Mirror 3: 0x1294af1 (stale) Mirror 4: 0x1294af1 (stale) Mirror 5: 0x1294af1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30cf6): Mirror 2: 0x1294af1 (stale) Mirror 3: 0x1294af1 (stale) Mirror 4: 0x1294af1 (stale) Mirror 5: 0x1294af1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30cf6: skipped 000286 1786827601.753754 mapwrite 0x02958e thru 0x035a1b (0xc48e bytes) 000288 1786827601.849342 MIRROR_RESYNC 5 mirrors 000290 1786827602.986008 write 0x015be0 thru 0x020b07 (0xaf28 bytes) 000291 1786827603.105681 trunc from 0x035a1c to 0x032f73 (0x2aa8 bytes) 000294 1786827603.165424 trunc from 0x032f74 to 0x02902d (0x9f46 bytes) 000297 1786827603.198025 mapwrite 0x028026 thru 0x02e337 (0x6312 bytes) 000298 1786827603.408211 MIRROR_RESYNC 5 mirrors 000301 1786827604.423557 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) 000302 1786827604.711962 write 0x00d02c thru 0x01adfa (0xddcf bytes) 000303 1786827604.802936 write 0x039fe6 thru 0x03ffff (0x601a bytes) 000304 1786827604.804981 mapwrite 0x009bad thru 0x00b6c9 (0x1b1d bytes) 000305 1786827604.934891 MIRROR_RESYNC 4 mirrors 000306 1786827606.139139 trunc from 0x040000 to 0x02dee2 (0x1211d bytes) 000307 1786827606.220945 mapread 0x016b8c thru 0x01a06e (0x34e3 bytes) 000309 1786827606.276839 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2dee3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dee3): Mirror 2: 0x9e0ca3eb (stale) Mirror 3: 0x9e0ca3eb (stale) Mirror 4: 0x9e0ca3eb (stale) CRC-32 checksum value for chunk [0, 0x2dee3): Mirror 1: 0x9e0ca3eb Mirror 5: 0x9e0ca3eb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dee3: skipped 000310 1786827607.417728 trunc from 0x02dee3 to 0x03304e (0x516c bytes) 000311 1786827607.583030 trunc from 0x03304f to 0x039b88 (0x6b3a bytes) 000313 1786827607.615686 write 0x032272 thru 0x034e9d (0x2c2c bytes) 000317 1786827607.632301 trunc from 0x039b89 to 0x005b4b (0x3403d bytes) 000318 1786827607.663191 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x5b4c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5b4c): Mirror 2: 0x814d5473 (stale) Mirror 3: 0x814d5473 (stale) Mirror 4: 0x814d5473 (stale) Mirror 5: 0x814d5473 (stale) CRC-32 checksum value for chunk [0, 0x5b4c): Mirror 1: 0x814d5473 Mirror 6: 0x814d5473 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5b4c: skipped 000320 1786827608.819935 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x5b4c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5b4c): Mirror 2: 0x814d5473 (stale) Mirror 3: 0x814d5473 (stale) Mirror 4: 0x814d5473 (stale) Mirror 5: 0x814d5473 (stale) CRC-32 checksum value for chunk [0, 0x5b4c): Mirror 1: 0x814d5473 Mirror 6: 0x814d5473 Mirror 7: 0x814d5473 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5b4c: skipped 000321 1786827609.850238 write 0x00c92a thru 0x010219 (0x38f0 bytes) 000322 1786827609.957817 mapwrite 0x0013d4 thru 0x006af0 (0x571d bytes) 000323 1786827610.005683 trunc from 0x01021a to 0x018f3f (0x8d26 bytes) 000324 1786827610.048613 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x18f40): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18f40): Mirror 2: 0x4c90d114 (stale) Mirror 3: 0x4c90d114 (stale) Mirror 4: 0x4c90d114 (stale) Mirror 5: 0x4c90d114 (stale) Mirror 6: 0x94f48332 (stale) Mirror 7: 0x94f48332 (stale) CRC-32 checksum value for chunk [0, 0x18f40): Mirror 1: 0x8f79db64 Mirror 8: 0x8f79db64 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18f40: skipped 000327 1786827611.042503 read 0x004ec3 thru 0x011d52 (0xce90 bytes) 000330 1786827611.122898 write 0x00da5b thru 0x017496 (0x9a3c bytes) 000331 1786827611.240414 read 0x00d5f2 thru 0x00e20d (0xc1c bytes) 000332 1786827611.241164 mapwrite 0x01511a thru 0x0157f5 (0x6dc bytes) 000333 1786827611.282650 read 0x00d62d thru 0x018f3f (0xb913 bytes) 000334 1786827611.293195 read 0x017800 thru 0x018f3f (0x1740 bytes) 000338 1786827611.295541 read 0x0035c2 thru 0x0133c2 (0xfe01 bytes) 000339 1786827611.297177 write 0x0303e0 thru 0x032a6d (0x268e bytes) 000342 1786827611.300073 mapread 0x00b5c7 thru 0x012ed6 (0x7910 bytes) 000343 1786827611.301761 mapwrite 0x039e4d thru 0x03a248 (0x3fc bytes) 000346 1786827611.384431 trunc from 0x03a249 to 0x03e224 (0x3fdc bytes) 000347 1786827611.404854 mapwrite 0x00e0f3 thru 0x01d6be (0xf5cc bytes) 000348 1786827611.448272 mapwrite 0x03e359 thru 0x03ffff (0x1ca7 bytes) 000349 1786827611.513777 mapwrite 0x032f82 thru 0x0397d9 (0x6858 bytes) 000351 1786827611.544369 read 0x01de52 thru 0x02034e (0x24fd bytes) 000352 1786827611.545244 read 0x01ac6b thru 0x02850b (0xd8a1 bytes) 000353 1786827611.545767 read 0x021772 thru 0x02a5c9 (0x8e58 bytes) 000355 1786827611.546838 read 0x0007a4 thru 0x005d9d (0x55fa bytes) 000357 1786827611.548633 write 0x026a05 thru 0x0336d6 (0xccd2 bytes) 000358 1786827611.550920 mapwrite 0x007a2e thru 0x014cc6 (0xd299 bytes) 000362 1786827611.593258 write 0x033dad thru 0x03b3e3 (0x7637 bytes) 000364 1786827611.595814 trunc from 0x040000 to 0x012b81 (0x2d47e bytes) 000368 1786827611.625046 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12b82): Mirror 3: 0x92251538 (stale) Mirror 4: 0x92251538 (stale) Mirror 5: 0x92251538 (stale) Mirror 6: 0xc40f79c5 (stale) Mirror 7: 0xc40f79c5 (stale) Mirror 8: 0xd7bac6ad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12b82): Mirror 3: 0x92251538 (stale) Mirror 4: 0x92251538 (stale) Mirror 5: 0x92251538 (stale) Mirror 6: 0xc40f79c5 (stale) Mirror 7: 0xc40f79c5 (stale) Mirror 8: 0xd7bac6ad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12b82: skipped 000369 1786827612.044885 trunc from 0x012b82 to 0x028382 (0x15801 bytes) 000370 1786827612.066405 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28383): Mirror 3: 0x63bb33c3 (stale) Mirror 4: 0x63bb33c3 (stale) Mirror 6: 0xde671a39 (stale) Mirror 7: 0xde671a39 (stale) Mirror 8: 0x4e09ba33 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28383): Mirror 3: 0x63bb33c3 (stale) Mirror 4: 0x63bb33c3 (stale) Mirror 6: 0xde671a39 (stale) Mirror 7: 0xde671a39 (stale) Mirror 8: 0x4e09ba33 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28383: skipped 000371 1786827612.470177 read 0x005fc8 thru 0x00f666 (0x969f bytes) 000375 1786827612.486409 write 0x03f9be thru 0x03ffff (0x642 bytes) 000376 1786827612.487807 mapread 0x01896b thru 0x02115a (0x87f0 bytes) 000377 1786827612.508072 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6304b0a5 (stale) Mirror 4: 0x6304b0a5 (stale) Mirror 6: 0xee94f030 (stale) Mirror 7: 0xee94f030 (stale) Mirror 8: 0x403e8264 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7b640f6e Mirror 9: 0x7b640f6e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000381 1786827613.398184 write 0x02252e thru 0x027033 (0x4b06 bytes) 000382 1786827613.500619 mapread 0x034eab thru 0x03ba91 (0x6be7 bytes) 000383 1786827613.522122 trunc from 0x040000 to 0x02e210 (0x11def bytes) 000384 1786827613.568413 trunc from 0x02e211 to 0x00b325 (0x22eeb bytes) 000385 1786827613.594422 mapwrite 0x023adf thru 0x031baf (0xe0d1 bytes) 000386 1786827613.688003 mapwrite 0x018aa5 thru 0x026b94 (0xe0f0 bytes) 000388 1786827613.796569 write 0x033598 thru 0x03ffff (0xca68 bytes) 000390 1786827613.805074 trunc from 0x040000 to 0x00c937 (0x336c8 bytes) 000392 1786827613.844534 mapread 0x001746 thru 0x006ef5 (0x57b0 bytes) 000396 1786827613.870402 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc938): Mirror 3: 0xcd103c51 (stale) Mirror 4: 0xcd103c51 (stale) Mirror 7: 0xa606b313 (stale) Mirror 8: 0xd9bc2860 (stale) Mirror 9: 0xf6d5cc8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc938): Mirror 3: 0xcd103c51 (stale) Mirror 4: 0xcd103c51 (stale) Mirror 7: 0xa606b313 (stale) Mirror 8: 0xd9bc2860 (stale) Mirror 9: 0xf6d5cc8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc938: skipped 000399 1786827614.261983 MIRROR_RESYNC 6 mirrors 000400 1786827615.694066 trunc from 0x00c938 to 0x02365a (0x16d23 bytes) 000401 1786827615.787826 mapread 0x01ac1e thru 0x0210c6 (0x64a9 bytes) 000402 1786827615.822147 trunc from 0x02365b to 0x03de9a (0x1a840 bytes) 000403 1786827615.852736 trunc from 0x03de9b to 0x0240dd (0x19dbd bytes) 000404 1786827615.879050 write 0x01cade thru 0x026869 (0x9d8c bytes) 000406 1786827615.887184 mapwrite 0x03a47e thru 0x03fe05 (0x5988 bytes) 000408 1786827616.022347 mapwrite 0x02389c thru 0x02c041 (0x87a6 bytes) 000409 1786827616.104337 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3fe06): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe06): Mirror 3: 0x9ee8337f (stale) Mirror 4: 0x9ee8337f (stale) Mirror 7: 0x9ee8337f (stale) Mirror 8: 0x9ee8337f (stale) Mirror 9: 0x9ee8337f (stale) CRC-32 checksum value for chunk [0, 0x3fe06): Mirror 1: 0xc2eca81e Mirror 10: 0xc2eca81e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe06: skipped 000411 1786827617.115806 mapwrite 0x012ce9 thru 0x01bf20 (0x9238 bytes) 000412 1786827617.370123 mapread 0x0015cf thru 0x002216 (0xc48 bytes) 000413 1786827617.392669 read 0x03f11f thru 0x03fe05 (0xce7 bytes) 000414 1786827617.398040 mapread 0x036f24 thru 0x03ba02 (0x4adf bytes) 000415 1786827617.421662 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fe06): Mirror 3: 0x9ee8337f (stale) Mirror 4: 0x9ee8337f (stale) Mirror 7: 0x9ee8337f (stale) Mirror 8: 0x9ee8337f (stale) Mirror 9: 0x9ee8337f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe06): Mirror 3: 0x9ee8337f (stale) Mirror 4: 0x9ee8337f (stale) Mirror 7: 0x9ee8337f (stale) Mirror 8: 0x9ee8337f (stale) Mirror 9: 0x9ee8337f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe06: skipped 000417 1786827617.825996 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fe06): Mirror 4: 0x9ee8337f (stale) Mirror 7: 0x9ee8337f (stale) Mirror 8: 0x9ee8337f (stale) Mirror 9: 0x9ee8337f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe06): Mirror 4: 0x9ee8337f (stale) Mirror 7: 0x9ee8337f (stale) Mirror 8: 0x9ee8337f (stale) Mirror 9: 0x9ee8337f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe06: skipped 000418 1786827618.141023 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fe06): Mirror 4: 0x9ee8337f (stale) Mirror 7: 0x9ee8337f (stale) Mirror 9: 0x9ee8337f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe06): Mirror 4: 0x9ee8337f (stale) Mirror 7: 0x9ee8337f (stale) Mirror 9: 0x9ee8337f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe06: skipped 000419 1786827618.656345 mapread 0x0099af thru 0x0176d0 (0xdd22 bytes) 000421 1786827618.700020 read 0x01b0bc thru 0x0286eb (0xd630 bytes) 000422 1786827618.716319 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fe06): Mirror 7: 0x9ee8337f (stale) Mirror 9: 0x9ee8337f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe06): Mirror 7: 0x9ee8337f (stale) Mirror 9: 0x9ee8337f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe06: skipped 000423 1786827618.992453 mapwrite 0x01978f thru 0x026814 (0xd086 bytes) 000424 1786827619.059800 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fe06): Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe06): Mirror 7: 0x9ee8337f (stale) Mirror 9: 0x9ee8337f (stale) CRC-32 checksum value for chunk [0, 0x3fe06): Mirror 1: 0x3e80ffa Mirror 10: 0x3e80ffa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe06: skipped 000425 1786827619.816955 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fe06): Mirror 9: 0x9ee8337f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe06): Mirror 9: 0x9ee8337f (stale) CRC-32 checksum value for chunk [0, 0x3fe06): Mirror 1: 0x3e80ffa Mirror 10: 0x3e80ffa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe06: skipped 000426 1786827620.467918 mapwrite 0x011fb0 thru 0x021c88 (0xfcd9 bytes) 000429 1786827620.692137 mapread 0x03d3c8 thru 0x03fe05 (0x2a3e bytes) 000430 1786827620.704617 trunc from 0x03fe06 to 0x0377c1 (0x8644 bytes) 000431 1786827620.737482 trunc from 0x0377c2 to 0x00167b (0x36146 bytes) 000432 1786827620.759817 write 0x003042 thru 0x004b64 (0x1b23 bytes) 000433 1786827620.766311 MIRROR_RESYNC 3 mirrors 000435 1786827621.501814 trunc from 0x004b65 to 0x02fcb3 (0x2b14f bytes) 000438 1786827621.594661 write 0x0201ee thru 0x024584 (0x4397 bytes) 000439 1786827621.606254 trunc from 0x02fcb4 to 0x03865b (0x89a8 bytes) 000440 1786827621.641211 MIRROR_RESYNC 3 mirrors 000441 1786827622.388237 mapread 0x02c179 thru 0x02cb77 (0x9ff bytes) 000442 1786827622.392165 mapread 0x01e6d9 thru 0x02ae16 (0xc73e bytes) 000443 1786827622.419958 trunc from 0x03865c to 0x001d38 (0x36923 bytes) 000444 1786827622.481957 read 0x001945 thru 0x001d38 (0x3f4 bytes) 000445 1786827622.490756 write 0x013f28 thru 0x01fd6f (0xbe48 bytes) 000446 1786827622.492669 mapwrite 0x02bbd5 thru 0x0362d5 (0xa701 bytes) 000448 1786827622.585701 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x362d6): Mirror 10: 0x2a7c681c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x362d6): Mirror 10: 0x2a7c681c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x362d6: skipped 000449 1786827622.853526 MIRROR_RESYNC 2 mirrors 000453 1786827623.571882 read 0x02da3c thru 0x0362d5 (0x889a bytes) 000456 1786827623.586581 write 0x03ecc5 thru 0x03ffff (0x133b bytes) 000457 1786827623.641031 mapwrite 0x01d761 thru 0x020fba (0x385a bytes) 000463 1786827623.707366 mapread 0x017d6b thru 0x01fb5d (0x7df3 bytes) 000466 1786827623.722200 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) 000468 1786827623.844185 read 0x028086 thru 0x02b7ef (0x376a bytes) 000469 1786827623.851888 mapwrite 0x026c42 thru 0x02daff (0x6ebe bytes) 000470 1786827623.902272 MIRROR_EXTEND to 2 mirrors 000471 1786827624.536404 trunc from 0x040000 to 0x031ce9 (0xe316 bytes) 000474 1786827624.690940 MIRROR_RESYNC 2 mirrors 000476 1786827625.316884 mapread 0x028af9 thru 0x031ce9 (0x91f1 bytes) 000477 1786827625.342201 write 0x031c6c thru 0x036304 (0x4699 bytes) 000478 1786827625.429745 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) 000479 1786827625.624494 mapwrite 0x0330ec thru 0x03c61b (0x9530 bytes) 000480 1786827625.727674 write 0x001879 thru 0x004f2e (0x36b6 bytes) 000481 1786827625.738036 write 0x014d01 thru 0x01fa6e (0xad6e bytes) 000482 1786827625.748819 read 0x02fbb8 thru 0x03c61b (0xca64 bytes) 000483 1786827625.749695 write 0x0118db thru 0x01a5d6 (0x8cfc bytes) 000486 1786827625.756730 write 0x000085 thru 0x00bbfa (0xbb76 bytes) 000489 1786827625.769186 mapwrite 0x02bc9e thru 0x035dc5 (0xa128 bytes) 000491 1786827625.852606 write 0x03b698 thru 0x03ffff (0x4968 bytes) 000494 1786827625.855246 write 0x02821a thru 0x035e58 (0xdc3f bytes) 000496 1786827625.858027 mapread 0x02189a thru 0x023f66 (0x26cd bytes) 000501 1786827625.875230 write 0x0282cb thru 0x02fe29 (0x7b5f bytes) 000502 1786827625.876717 trunc from 0x040000 to 0x0341a2 (0xbe5d bytes) 000505 1786827625.898301 read 0x025f3a thru 0x0341a2 (0xe269 bytes) 000507 1786827625.899763 mapwrite 0x0092c5 thru 0x011d1a (0x8a56 bytes) 000508 1786827625.947860 mapwrite 0x01ab54 thru 0x026900 (0xbdad bytes) 000509 1786827625.999512 read 0x01382b thru 0x018fe1 (0x57b7 bytes) 000512 1786827626.003613 mapwrite 0x02a348 thru 0x02bc15 (0x18ce bytes) 000513 1786827626.046829 write 0x00f2ba thru 0x01a0df (0xae26 bytes) 000514 1786827626.049379 write 0x0225a9 thru 0x03114a (0xeba2 bytes) 000515 1786827626.051800 mapread 0x03130b thru 0x032d0f (0x1a05 bytes) 000516 1786827626.054385 trunc from 0x0341a3 to 0x004c2e (0x2f574 bytes) 000518 1786827626.073593 mapread 0x0027e3 thru 0x00287e (0x9c bytes) 000519 1786827626.074720 mapread 0x002692 thru 0x0030f6 (0xa65 bytes) 000520 1786827626.077048 trunc from 0x004c2f to 0x000948 (0x42e6 bytes) 000522 1786827626.123551 MIRROR_EXTEND to 2 mirrors 000523 1786827626.724601 mapwrite 0x029fc2 thru 0x034882 (0xa8c1 bytes) 000524 1786827626.949503 mapread 0x033697 thru 0x034882 (0x11ec bytes) 000525 1786827626.950548 read 0x00ec7f thru 0x019b24 (0xaea6 bytes) 000526 1786827626.960855 trunc from 0x034883 to 0x01c8bd (0x17fc5 bytes) 000527 1786827626.988751 MIRROR_RESYNC 2 mirrors 000529 1786827627.611893 trunc from 0x01c8be to 0x014677 (0x8246 bytes) 000530 1786827627.755631 read 0x00c64c thru 0x010130 (0x3ae5 bytes) 000531 1786827627.767294 write 0x0359f0 thru 0x03ffff (0xa610 bytes) 000532 1786827627.770021 trunc from 0x040000 to 0x02e800 (0x117ff bytes) 000534 1786827627.800873 read 0x029279 thru 0x02e800 (0x5588 bytes) 000536 1786827627.812394 read 0x024633 thru 0x02a0d1 (0x5a9f bytes) 000537 1786827627.818395 MIRROR_RESYNC 2 mirrors 000540 1786827628.493122 read 0x005ca2 thru 0x00ccf1 (0x7050 bytes) 000541 1786827628.506003 mapwrite 0x02f096 thru 0x034a55 (0x59c0 bytes) 000542 1786827628.662394 read 0x01de57 thru 0x022e83 (0x502d bytes) 000543 1786827628.671293 write 0x038cc2 thru 0x03c27e (0x35bd bytes) 000545 1786827628.674927 read 0x022bd6 thru 0x02b8ba (0x8ce5 bytes) 000546 1786827628.675974 trunc from 0x03c27f to 0x02ead1 (0xd7ad bytes) 000547 1786827628.696891 read 0x02d487 thru 0x02ead1 (0x164b bytes) 000548 1786827628.704085 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) 000550 1786827628.857835 mapwrite 0x021759 thru 0x025ed4 (0x477c bytes) 000551 1786827628.945798 read 0x02d49c thru 0x02ead1 (0x1636 bytes) 000552 1786827628.951544 trunc from 0x02ead2 to 0x02bf3e (0x2b93 bytes) 000554 1786827628.970803 read 0x000b5e thru 0x006286 (0x5729 bytes) 000556 1786827628.983057 write 0x001c08 thru 0x0047d3 (0x2bcc bytes) 000558 1786827628.984926 write 0x02c26b thru 0x031ffe (0x5d94 bytes) 000559 1786827628.986267 read 0x00da14 thru 0x0127b5 (0x4da2 bytes) 000560 1786827628.987013 mapwrite 0x004ab9 thru 0x012aac (0xdff4 bytes) 000561 1786827629.048132 trunc from 0x031fff to 0x0310df (0xf1f bytes) 000563 1786827629.072525 trunc from 0x0310e0 to 0x03bcca (0xabeb bytes) 000565 1786827629.096782 write 0x003b48 thru 0x00dbd4 (0xa08d bytes) 000567 1786827629.100731 read 0x01bfb2 thru 0x0292ce (0xd31d bytes) 000568 1786827629.102555 mapwrite 0x027677 thru 0x02d639 (0x5fc3 bytes) 000571 1786827629.160294 mapwrite 0x000bf1 thru 0x0099a0 (0x8db0 bytes) 000573 1786827629.213613 mapwrite 0x017543 thru 0x01f89a (0x8358 bytes) 000575 1786827629.312657 write 0x00cb09 thru 0x01ad6c (0xe264 bytes) 000579 1786827629.318913 trunc from 0x03bccb to 0x020013 (0x1bcb7 bytes) 000581 1786827629.341318 mapread 0x0042ca thru 0x01085b (0xc592 bytes) 000583 1786827629.345709 write 0x0203ea thru 0x024755 (0x436c bytes) 000584 1786827629.348030 write 0x03ce48 thru 0x03ffff (0x31b8 bytes) 000589 1786827629.357760 trunc from 0x040000 to 0x00ee21 (0x311de bytes) 000593 1786827629.383840 mapwrite 0x01abd0 thru 0x021736 (0x6b67 bytes) 000594 1786827629.479064 read 0x01a467 thru 0x021736 (0x72d0 bytes) 000596 1786827629.482149 trunc from 0x021737 to 0x00e5c9 (0x1316d bytes) 000597 1786827629.510836 trunc from 0x00e5ca to 0x0393fc (0x2ae33 bytes) 000601 1786827629.530907 mapwrite 0x021e82 thru 0x026e83 (0x5002 bytes) 000602 1786827629.595440 read 0x0148a2 thru 0x01fe7c (0xb5db bytes) 000603 1786827629.599141 write 0x02dfb2 thru 0x0376df (0x972e bytes) 000608 1786827629.606835 write 0x00941b thru 0x00c5b4 (0x319a bytes) 000609 1786827629.608134 mapwrite 0x0175dc thru 0x02122b (0x9c50 bytes) 000610 1786827629.680843 read 0x0117fb thru 0x013c2f (0x2435 bytes) 000612 1786827629.689259 read 0x009409 thru 0x016ba1 (0xd799 bytes) 000613 1786827629.690021 trunc from 0x0393fd to 0x036b37 (0x28c5 bytes) 000615 1786827629.720358 write 0x001440 thru 0x00cabb (0xb67c bytes) 000616 1786827629.722762 trunc from 0x036b38 to 0x01bcfe (0x1ae39 bytes) 000618 1786827629.738905 mapread 0x009bc8 thru 0x00d403 (0x383c bytes) 000623 1786827629.751821 mapwrite 0x0119e2 thru 0x01987e (0x7e9d bytes) 000625 1786827629.831144 mapwrite 0x0285bc thru 0x02fbbc (0x7601 bytes) 000629 1786827629.955686 mapread 0x025500 thru 0x02fbbc (0xa6bd bytes) 000632 1786827629.962902 mapwrite 0x0063b0 thru 0x006df7 (0xa48 bytes) 000633 1786827630.032165 write 0x036391 thru 0x03ffff (0x9c6f bytes) 000634 1786827630.034735 mapwrite 0x0055c4 thru 0x01243d (0xce7a bytes) 000635 1786827630.127166 write 0x01b0f1 thru 0x01f6a8 (0x45b8 bytes) 000637 1786827630.130575 mapwrite 0x015dc3 thru 0x019493 (0x36d1 bytes) 000639 1786827630.195900 mapread 0x01be30 thru 0x028ccb (0xce9c bytes) 000640 1786827630.198532 mapwrite 0x03ad1a thru 0x03e7c9 (0x3ab0 bytes) 000641 1786827630.242011 mapwrite 0x021b1c thru 0x02d761 (0xbc46 bytes) 000642 1786827630.320597 MIRROR_EXTEND to 2 mirrors 000643 1786827631.188383 trunc from 0x040000 to 0x02008b (0x1ff74 bytes) 000644 1786827631.291746 write 0x0076d0 thru 0x016f64 (0xf895 bytes) 000645 1786827631.302984 write 0x02b3e5 thru 0x02cb59 (0x1775 bytes) 000648 1786827631.306655 read 0x0142f3 thru 0x01f603 (0xb311 bytes) 000651 1786827631.320894 mapwrite 0x01dbb3 thru 0x0250d7 (0x7525 bytes) 000653 1786827631.402250 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2cb5a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cb5a): Mirror 2: 0x616b0a1c (stale) CRC-32 checksum value for chunk [0, 0x2cb5a): Mirror 1: 0x2e35b525 Mirror 3: 0x2e35b525 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cb5a: skipped 000654 1786827632.163261 write 0x00df04 thru 0x012a1c (0x4b19 bytes) 000655 1786827632.281506 read 0x02bef9 thru 0x02cb59 (0xc61 bytes) 000656 1786827632.293730 MIRROR_RESYNC 3 mirrors 000657 1786827633.036812 mapread 0x013218 thru 0x022263 (0xf04c bytes) 000659 1786827633.086229 trunc from 0x02cb5a to 0x00f0bb (0x1da9e bytes) 000660 1786827633.194678 write 0x023e44 thru 0x0274cf (0x368c bytes) 000662 1786827633.199597 trunc from 0x0274d0 to 0x001cab (0x25824 bytes) 000666 1786827633.230281 mapwrite 0x0248ca thru 0x02ce04 (0x853b bytes) 000667 1786827633.349665 trunc from 0x02ce05 to 0x00e45d (0x1e9a7 bytes) 000668 1786827633.389811 mapread 0x002b86 thru 0x00e45d (0xb8d8 bytes) 000669 1786827633.415024 mapwrite 0x032af0 thru 0x039349 (0x685a bytes) 000671 1786827633.561725 read 0x006718 thru 0x014548 (0xde31 bytes) 000672 1786827633.577350 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3934a): Mirror 2: 0xd0fcd1aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3934a): Mirror 2: 0xd0fcd1aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3934a: skipped 000674 1786827633.826490 read 0x00e68d thru 0x0116ee (0x3062 bytes) 000676 1786827633.832966 mapwrite 0x020bc3 thru 0x023da9 (0x31e7 bytes) 000680 1786827633.927718 mapread 0x01bcda thru 0x02997d (0xdca4 bytes) 000682 1786827633.931545 write 0x003fcd thru 0x005ad9 (0x1b0d bytes) 000684 1786827633.948898 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3934a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3934a): Mirror 2: 0xd0fcd1aa (stale) CRC-32 checksum value for chunk [0, 0x3934a): Mirror 1: 0xe33cad1f Mirror 3: 0xe33cad1f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3934a: skipped 000687 1786827634.921974 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3934a): Mirror 2: 0xd0fcd1aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3934a): Mirror 2: 0xd0fcd1aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3934a: skipped 000688 1786827635.205089 mapwrite 0x016e0b thru 0x018a71 (0x1c67 bytes) 000690 1786827635.326466 trunc from 0x03934a to 0x039141 (0x208 bytes) 000692 1786827635.353474 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) 000693 1786827635.501071 mapwrite 0x026b9e thru 0x02d908 (0x6d6b bytes) 000694 1786827635.576987 read 0x024fe4 thru 0x02cc26 (0x7c43 bytes) 000695 1786827635.577759 mapread 0x02c0fa thru 0x039141 (0xd048 bytes) 000698 1786827635.592262 write 0x02f9e4 thru 0x030b98 (0x11b5 bytes) 000699 1786827635.597086 mapread 0x02d884 thru 0x02e3f1 (0xb6e bytes) 000702 1786827635.603535 trunc from 0x039142 to 0x03c4d8 (0x3397 bytes) 000703 1786827635.627400 read 0x0117d2 thru 0x01e0a9 (0xc8d8 bytes) 000705 1786827635.641048 read 0x0319a5 thru 0x03244a (0xaa6 bytes) 000706 1786827635.641704 write 0x012ed2 thru 0x01ad86 (0x7eb5 bytes) 000713 1786827635.656577 mapwrite 0x01eeb5 thru 0x0271a5 (0x82f1 bytes) 000714 1786827635.746069 write 0x02765d thru 0x02ad88 (0x372c bytes) 000716 1786827635.747860 mapwrite 0x02e87a thru 0x0321c6 (0x394d bytes) 000718 1786827635.880041 trunc from 0x03c4d9 to 0x036d26 (0x57b2 bytes) 000720 1786827635.917714 write 0x01bdf6 thru 0x02a9a9 (0xebb4 bytes) 000723 1786827635.923946 read 0x00be59 thru 0x01a9e2 (0xeb8a bytes) 000724 1786827635.924744 mapread 0x0327bc thru 0x036d26 (0x456b bytes) 000726 1786827635.932381 MIRROR_EXTEND to 2 mirrors 000727 1786827636.696586 MIRROR_EXTEND to 3 mirrors 000728 1786827637.513211 MIRROR_RESYNC 3 mirrors 000729 1786827637.826971 read 0x029789 thru 0x036d26 (0xd59e bytes) 000731 1786827637.847452 trunc from 0x036d27 to 0x008465 (0x2e8c1 bytes) 000734 1786827637.924182 write 0x02a4a0 thru 0x034ec2 (0xaa23 bytes) 000735 1786827637.930320 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x34ec3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34ec3): Mirror 2: 0xd7c8c00a (stale) Mirror 3: 0xd7c8c00a (stale) CRC-32 checksum value for chunk [0, 0x34ec3): Mirror 1: 0xfd3991e1 Mirror 4: 0xfd3991e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34ec3: skipped 000736 1786827638.647552 read 0x020ce1 thru 0x022791 (0x1ab1 bytes) 000737 1786827638.661751 trunc from 0x034ec3 to 0x028fbf (0xbf03 bytes) 000738 1786827638.739383 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x28fc0): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28fc0): Mirror 2: 0x7789db16 (stale) Mirror 3: 0x7789db16 (stale) Mirror 4: 0x5f3dacd4 (stale) CRC-32 checksum value for chunk [0, 0x28fc0): Mirror 1: 0x5f3dacd4 Mirror 5: 0x5f3dacd4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28fc0: skipped 000739 1786827639.502262 trunc from 0x028fc0 to 0x00f52a (0x19a95 bytes) 000740 1786827639.598286 write 0x01ea24 thru 0x027158 (0x8735 bytes) 000741 1786827639.606274 MIRROR_RESYNC 5 mirrors 000744 1786827640.651714 MIRROR_EXTEND to 6 mirrors 000745 1786827641.623421 mapwrite 0x03f288 thru 0x03ffff (0xd78 bytes) 000746 1786827641.753881 read 0x01ddb3 thru 0x0252a7 (0x74f5 bytes) 000747 1786827641.765306 write 0x016d49 thru 0x0194cf (0x2787 bytes) 000749 1786827641.775733 trunc from 0x040000 to 0x03d30a (0x2cf5 bytes) 000750 1786827641.798789 MIRROR_RESYNC 6 mirrors 000753 1786827643.070330 write 0x02363b thru 0x028f2a (0x58f0 bytes) 000754 1786827643.136600 read 0x029c36 thru 0x03777b (0xdb46 bytes) 000757 1786827643.148245 write 0x011f4d thru 0x021488 (0xf53c bytes) 000759 1786827643.160285 mapread 0x02a167 thru 0x02d3ec (0x3286 bytes) 000760 1786827643.162258 mapwrite 0x012a42 thru 0x013b3d (0x10fc bytes) 000762 1786827643.211985 read 0x008e45 thru 0x00c6c9 (0x3885 bytes) 000766 1786827643.226649 trunc from 0x03d30b to 0x019a1c (0x238ee bytes) 000768 1786827643.249271 write 0x0025d9 thru 0x00e990 (0xc3b8 bytes) 000769 1786827643.251803 mapread 0x0142a4 thru 0x019a1c (0x5779 bytes) 000771 1786827643.266846 trunc from 0x019a1d to 0x01f206 (0x57ea bytes) 000772 1786827643.284957 write 0x038aa3 thru 0x03d4c0 (0x4a1e bytes) 000773 1786827643.288610 mapread 0x02036c thru 0x023a73 (0x3708 bytes) 000776 1786827643.311466 mapwrite 0x03ab72 thru 0x03ffff (0x548e bytes) 000777 1786827643.368262 read 0x0032e1 thru 0x006e6e (0x3b8e bytes) 000778 1786827643.369246 mapwrite 0x013bfd thru 0x01c847 (0x8c4b bytes) 000779 1786827643.454021 read 0x01a288 thru 0x01b546 (0x12bf bytes) 000780 1786827643.455492 mapread 0x0151c7 thru 0x01b0e1 (0x5f1b bytes) 000781 1786827643.465833 mapwrite 0x039b78 thru 0x03e14d (0x45d6 bytes) 000782 1786827643.531426 write 0x00cb82 thru 0x019ead (0xd32c bytes) 000783 1786827643.534439 mapwrite 0x01600c thru 0x021bd1 (0xbbc6 bytes) 000784 1786827643.598981 write 0x02217b thru 0x02d3bc (0xb242 bytes) 000790 1786827643.609957 read 0x026f9e thru 0x034349 (0xd3ac bytes) 000792 1786827643.612893 trunc from 0x040000 to 0x0396f6 (0x6909 bytes) 000793 1786827643.633871 read 0x01ce46 thru 0x023b30 (0x6ceb bytes) 000794 1786827643.638582 trunc from 0x0396f7 to 0x038e19 (0x8dd bytes) 000795 1786827643.662639 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38e1a): Mirror 2: 0xa5e9f62d (stale) Mirror 3: 0xa5e9f62d (stale) Mirror 4: 0xa5e9f62d (stale) Mirror 5: 0xa5e9f62d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38e1a): Mirror 2: 0xa5e9f62d (stale) Mirror 3: 0xa5e9f62d (stale) Mirror 4: 0xa5e9f62d (stale) Mirror 5: 0xa5e9f62d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38e1a: skipped 000796 1786827644.200455 mapwrite 0x02e0b1 thru 0x03b40c (0xd35c bytes) 000797 1786827644.364519 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b40d): Mirror 3: 0x810859c6 (stale) Mirror 4: 0x810859c6 (stale) Mirror 5: 0x810859c6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b40d): Mirror 3: 0x810859c6 (stale) Mirror 4: 0x810859c6 (stale) Mirror 5: 0x810859c6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b40d: skipped 000799 1786827644.858964 MIRROR_RESYNC 4 mirrors 000800 1786827645.746437 MIRROR_EXTEND to 5 mirrors 000802 1786827646.500460 MIRROR_EXTEND to 6 mirrors 000804 1786827647.554626 trunc from 0x03b40d to 0x000d3c (0x3a6d0 bytes) 000806 1786827647.696603 mapwrite 0x03dc7a thru 0x03ffff (0x2386 bytes) 000807 1786827647.787546 trunc from 0x040000 to 0x00b4a7 (0x34b58 bytes) 000808 1786827647.803276 mapread 0x00538c thru 0x00b4a7 (0x611c bytes) 000811 1786827647.817861 mapwrite 0x00c3dc thru 0x011e24 (0x5a49 bytes) 000812 1786827647.938894 mapwrite 0x038b4e thru 0x03ffff (0x74b2 bytes) 000814 1786827648.031248 write 0x03972c thru 0x039b0a (0x3df bytes) 000817 1786827648.040152 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8c5357cb (stale) Mirror 4: 0x8c5357cb (stale) Mirror 5: 0x8c5357cb (stale) Mirror 6: 0x8c5357cb (stale) Mirror 7: 0x8c5357cb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb6dd8d86 Mirror 8: 0xb6dd8d86 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000819 1786827649.451426 mapwrite 0x030e1e thru 0x032510 (0x16f3 bytes) 000820 1786827649.626318 write 0x019a7d thru 0x028714 (0xec98 bytes) 000822 1786827649.642659 mapwrite 0x017655 thru 0x02595c (0xe308 bytes) 000823 1786827649.710808 mapwrite 0x0133a6 thru 0x01629a (0x2ef5 bytes) 000824 1786827649.755573 read 0x02167c thru 0x023b30 (0x24b5 bytes) 000825 1786827649.756980 mapread 0x03d852 thru 0x03ffff (0x27ae bytes) 000826 1786827649.776795 mapwrite 0x007906 thru 0x00c0a2 (0x479d bytes) 000827 1786827649.861014 read 0x0252dc thru 0x02c6fe (0x7423 bytes) 000828 1786827649.870748 write 0x03bf29 thru 0x03ffff (0x40d7 bytes) 000832 1786827649.875931 mapwrite 0x011eba thru 0x017fd4 (0x611b bytes) 000834 1786827649.965588 write 0x0338ae thru 0x03f6c2 (0xbe15 bytes) 000835 1786827649.967646 read 0x01a9a6 thru 0x023fcf (0x962a bytes) 000837 1786827649.969254 read 0x0128b4 thru 0x02021b (0xd968 bytes) 000838 1786827649.972942 MIRROR_RESYNC 7 mirrors 000843 1786827651.259596 trunc from 0x040000 to 0x00963d (0x369c2 bytes) 000844 1786827651.340202 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x963e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x963e): Mirror 3: 0xd9589b06 (stale) Mirror 4: 0xd9589b06 (stale) Mirror 5: 0xd9589b06 (stale) Mirror 6: 0xd9589b06 (stale) Mirror 7: 0xd9589b06 (stale) Mirror 8: 0xd9589b06 (stale) CRC-32 checksum value for chunk [0, 0x963e): Mirror 1: 0xd9589b06 Mirror 9: 0xd9589b06 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x963e: skipped 000845 1786827652.304950 mapread 0x0070f1 thru 0x00963d (0x254d bytes) 000846 1786827652.355716 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x963e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x963e): Mirror 3: 0xd9589b06 (stale) Mirror 4: 0xd9589b06 (stale) Mirror 5: 0xd9589b06 (stale) Mirror 6: 0xd9589b06 (stale) Mirror 7: 0xd9589b06 (stale) Mirror 8: 0xd9589b06 (stale) CRC-32 checksum value for chunk [0, 0x963e): Mirror 1: 0xd9589b06 Mirror 9: 0xd9589b06 Mirror 10: 0xd9589b06 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x963e: skipped 000847 1786827653.345731 read 0x006444 thru 0x00963d (0x31fa bytes) 000848 1786827653.363941 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x963e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x963e): Mirror 3: 0xd9589b06 (stale) Mirror 4: 0xd9589b06 (stale) Mirror 5: 0xd9589b06 (stale) Mirror 6: 0xd9589b06 (stale) Mirror 7: 0xd9589b06 (stale) Mirror 8: 0xd9589b06 (stale) CRC-32 checksum value for chunk [0, 0x963e): Mirror 1: 0xd9589b06 Mirror 9: 0xd9589b06 Mirror 10: 0xd9589b06 Mirror 11: 0xd9589b06 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x963e: skipped 000850 1786827654.388759 trunc from 0x00963e to 0x029ce7 (0x206aa bytes) 000851 1786827654.480729 mapread 0x023906 thru 0x029ce7 (0x63e2 bytes) 000852 1786827654.504470 write 0x015656 thru 0x017476 (0x1e21 bytes) 000857 1786827654.529971 write 0x039635 thru 0x03ffff (0x69cb bytes) 000859 1786827654.537349 read 0x034e37 thru 0x03ffff (0xb1c9 bytes) 000862 1786827654.566310 mapread 0x03c2de thru 0x03ffff (0x3d22 bytes) 000863 1786827654.594921 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6ebd230a (stale) Mirror 4: 0x6ebd230a (stale) Mirror 5: 0x6ebd230a (stale) Mirror 6: 0x6ebd230a (stale) Mirror 7: 0x6ebd230a (stale) Mirror 8: 0x6ebd230a (stale) Mirror 9: 0x36f9e703 (stale) Mirror 11: 0x36f9e703 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6ebd230a (stale) Mirror 4: 0x6ebd230a (stale) Mirror 5: 0x6ebd230a (stale) Mirror 6: 0x6ebd230a (stale) Mirror 7: 0x6ebd230a (stale) Mirror 8: 0x6ebd230a (stale) Mirror 9: 0x36f9e703 (stale) Mirror 11: 0x36f9e703 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000865 1786827655.281463 mapwrite 0x00d916 thru 0x014473 (0x6b5e bytes) 000868 1786827655.339952 write 0x0211d0 thru 0x02f4ad (0xe2de bytes) 000873 1786827655.358062 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6ebd230a (stale) Mirror 4: 0x6ebd230a (stale) Mirror 5: 0x6ebd230a (stale) Mirror 6: 0x6ebd230a (stale) Mirror 8: 0x6ebd230a (stale) Mirror 9: 0x36f9e703 (stale) Mirror 11: 0x36f9e703 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6ebd230a (stale) Mirror 4: 0x6ebd230a (stale) Mirror 5: 0x6ebd230a (stale) Mirror 6: 0x6ebd230a (stale) Mirror 8: 0x6ebd230a (stale) Mirror 9: 0x36f9e703 (stale) Mirror 11: 0x36f9e703 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000874 1786827655.803544 mapwrite 0x002e3f thru 0x003041 (0x203 bytes) 000875 1786827655.869831 mapread 0x00a5ba thru 0x0129f7 (0x843e bytes) 000876 1786827655.884488 trunc from 0x040000 to 0x03af65 (0x509a bytes) 000878 1786827655.907116 mapwrite 0x029c7b thru 0x033367 (0x96ed bytes) 000879 1786827655.992984 read 0x00bb08 thru 0x0182ea (0xc7e3 bytes) 000880 1786827656.011824 trunc from 0x03af66 to 0x005679 (0x358ec bytes) 000881 1786827656.038660 trunc from 0x00567a to 0x015509 (0xfe90 bytes) 000882 1786827656.056611 mapwrite 0x017c58 thru 0x01c4df (0x4888 bytes) truncating to largest ever: 0x3e8e9 000883 1786827656.168447 trunc from 0x01c4e0 to 0x03e8e8 (0x22409 bytes) 000884 1786827656.202061 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3e8e9): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e8e9): Mirror 3: 0xab0311f5 (stale) Mirror 4: 0xab0311f5 (stale) Mirror 5: 0xab0311f5 (stale) Mirror 6: 0xab0311f5 (stale) Mirror 8: 0xab0311f5 (stale) Mirror 9: 0x9111644e (stale) Mirror 11: 0x9111644e (stale) CRC-32 checksum value for chunk [0, 0x3e8e9): Mirror 1: 0x1a52f48 Mirror 12: 0x1a52f48 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e8e9: skipped 000886 1786827657.571663 mapwrite 0x01443d thru 0x01e07b (0x9c3f bytes) 000887 1786827657.827929 read 0x0218bc thru 0x03172e (0xfe73 bytes) 000889 1786827657.865628 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3e8e9): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e8e9): Mirror 3: 0xab0311f5 (stale) Mirror 4: 0xab0311f5 (stale) Mirror 5: 0xab0311f5 (stale) Mirror 6: 0xab0311f5 (stale) Mirror 8: 0xab0311f5 (stale) Mirror 9: 0x9111644e (stale) Mirror 11: 0x9111644e (stale) Mirror 12: 0x1a52f48 (stale) CRC-32 checksum value for chunk [0, 0x3e8e9): Mirror 1: 0xe52dc50e Mirror 13: 0xe52dc50e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e8e9: skipped 000892 1786827658.980849 write 0x00f0e6 thru 0x00f5ac (0x4c7 bytes) 000896 1786827659.060486 write 0x019149 thru 0x0269e9 (0xd8a1 bytes) 000899 1786827659.074321 read 0x02f32a thru 0x03425a (0x4f31 bytes) 000905 1786827659.107543 MIRROR_RESYNC 10 mirrors 000906 1786827660.988400 mapwrite 0x01196c thru 0x0170ea (0x577f bytes) 000908 1786827661.257928 mapwrite 0x0001eb thru 0x001b04 (0x191a bytes) 000909 1786827661.307190 read 0x03da20 thru 0x03e8e8 (0xec9 bytes) 000910 1786827661.314449 mapwrite 0x0020d1 thru 0x00394d (0x187d bytes) 000911 1786827661.354043 mapwrite 0x03ceb0 thru 0x03ffff (0x3150 bytes) 000912 1786827661.418091 read 0x0200b3 thru 0x02adef (0xad3d bytes) 000913 1786827661.434755 mapread 0x0032b8 thru 0x00cd4c (0x9a95 bytes) 000914 1786827661.439760 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x562e5d58 (stale) Mirror 4: 0x562e5d58 (stale) Mirror 5: 0x562e5d58 (stale) Mirror 6: 0x562e5d58 (stale) Mirror 8: 0x562e5d58 (stale) Mirror 9: 0x562e5d58 (stale) Mirror 11: 0x562e5d58 (stale) Mirror 12: 0x562e5d58 (stale) Mirror 13: 0x562e5d58 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa74074ae Mirror 14: 0xa74074ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000915 1786827663.007401 write 0x022ccf thru 0x0240d5 (0x1407 bytes) 000918 1786827663.094797 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x562e5d58 (stale) Mirror 4: 0x562e5d58 (stale) Mirror 5: 0x562e5d58 (stale) Mirror 6: 0x562e5d58 (stale) Mirror 8: 0x562e5d58 (stale) Mirror 9: 0x562e5d58 (stale) Mirror 11: 0x562e5d58 (stale) Mirror 13: 0x562e5d58 (stale) Mirror 14: 0xa74074ae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x562e5d58 (stale) Mirror 4: 0x562e5d58 (stale) Mirror 5: 0x562e5d58 (stale) Mirror 6: 0x562e5d58 (stale) Mirror 8: 0x562e5d58 (stale) Mirror 9: 0x562e5d58 (stale) Mirror 11: 0x562e5d58 (stale) Mirror 13: 0x562e5d58 (stale) Mirror 14: 0xa74074ae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000919 1786827663.819221 mapread 0x00901a thru 0x0114e8 (0x84cf bytes) 000922 1786827663.858366 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x562e5d58 (stale) Mirror 4: 0x562e5d58 (stale) Mirror 5: 0x562e5d58 (stale) Mirror 6: 0x562e5d58 (stale) Mirror 8: 0x562e5d58 (stale) Mirror 9: 0x562e5d58 (stale) Mirror 11: 0x562e5d58 (stale) Mirror 13: 0x562e5d58 (stale) Mirror 14: 0xa74074ae (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9ae794c2 Mirror 15: 0x9ae794c2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000923 1786827665.166817 read 0x020aff thru 0x02ca7e (0xbf80 bytes) 000924 1786827665.200710 mapread 0x02ada7 thru 0x034ae3 (0x9d3d bytes) 000925 1786827665.245300 write 0x036876 thru 0x03ffff (0x978a bytes) 000929 1786827665.334273 trunc from 0x040000 to 0x02d87d (0x12782 bytes) 000930 1786827665.355939 trunc from 0x02d87e to 0x02490b (0x8f72 bytes) 000931 1786827665.368020 mapread 0x015387 thru 0x0183e7 (0x3061 bytes) 000932 1786827665.385521 write 0x0377dd thru 0x03d1f2 (0x5a16 bytes) 000933 1786827665.391672 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3d1f3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d1f3): Mirror 3: 0x5ac47931 (stale) Mirror 4: 0x5ac47931 (stale) Mirror 5: 0x5ac47931 (stale) Mirror 6: 0x5ac47931 (stale) Mirror 8: 0x5ac47931 (stale) Mirror 9: 0x5ac47931 (stale) Mirror 11: 0x5ac47931 (stale) Mirror 13: 0x5ac47931 (stale) Mirror 14: 0x892c2da7 (stale) Mirror 15: 0x69376f33 (stale) CRC-32 checksum value for chunk [0, 0x3d1f3): Mirror 1: 0xd067c1f8 Mirror 16: 0xd067c1f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d1f3: skipped 000940 1786827666.488430 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3d1f3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d1f3): Mirror 3: 0x5ac47931 (stale) Mirror 4: 0x5ac47931 (stale) Mirror 5: 0x5ac47931 (stale) Mirror 6: 0x5ac47931 (stale) Mirror 8: 0x5ac47931 (stale) Mirror 9: 0x5ac47931 (stale) Mirror 11: 0x5ac47931 (stale) Mirror 13: 0x5ac47931 (stale) Mirror 14: 0x892c2da7 (stale) Mirror 15: 0x69376f33 (stale) CRC-32 checksum value for chunk [0, 0x3d1f3): Mirror 1: 0xd067c1f8 Mirror 16: 0xd067c1f8 Mirror 17: 0xd067c1f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d1f3: skipped 000944 1786827667.830358 mapwrite 0x0366f6 thru 0x0384f9 (0x1e04 bytes) 000945 1786827668.108382 mapread 0x036986 thru 0x03cad4 (0x614f bytes) 000946 1786827668.112088 trunc from 0x03d1f3 to 0x030f46 (0xc2ac bytes) 000952 1786827668.169497 read 0x011aef thru 0x0134c4 (0x19d6 bytes) 000958 1786827668.187804 MIRROR_RESYNC 13 mirrors 000959 1786827670.343644 mapread 0x006a28 thru 0x015446 (0xea1f bytes) 000961 1786827670.370575 write 0x032ee1 thru 0x0351e9 (0x2309 bytes) 000963 1786827670.441641 write 0x007bc9 thru 0x0081fb (0x633 bytes) 000964 1786827670.454558 MIRROR_RESYNC 13 mirrors 000965 1786827672.702047 MIRROR_SPLIT mirror 12 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) 000968 1786827673.187936 trunc from 0x0351ea to 0x024895 (0x10954 bytes) 000969 1786827673.410204 mapread 0x014ddb thru 0x020e7d (0xc0a3 bytes) 000970 1786827673.436316 write 0x025008 thru 0x02ed2a (0x9d23 bytes) 000971 1786827673.441452 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x2ed2b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ed2b): Mirror 3: 0x19acad49 (stale) Mirror 4: 0x19acad49 (stale) Mirror 5: 0x19acad49 (stale) Mirror 6: 0x19acad49 (stale) Mirror 8: 0x19acad49 (stale) Mirror 9: 0x19acad49 (stale) Mirror 11: 0x19acad49 (stale) Mirror 13: 0x19acad49 (stale) Mirror 14: 0x19acad49 (stale) Mirror 15: 0x19acad49 (stale) Mirror 16: 0x19acad49 (stale) CRC-32 checksum value for chunk [0, 0x2ed2b): Mirror 1: 0xb4cad35f Mirror 17: 0xb4cad35f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ed2b: skipped 000972 1786827674.328659 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ed2b): Mirror 4: 0x19acad49 (stale) Mirror 5: 0x19acad49 (stale) Mirror 6: 0x19acad49 (stale) Mirror 8: 0x19acad49 (stale) Mirror 9: 0x19acad49 (stale) Mirror 11: 0x19acad49 (stale) Mirror 13: 0x19acad49 (stale) Mirror 14: 0x19acad49 (stale) Mirror 15: 0x19acad49 (stale) Mirror 16: 0x19acad49 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ed2b): Mirror 4: 0x19acad49 (stale) Mirror 5: 0x19acad49 (stale) Mirror 6: 0x19acad49 (stale) Mirror 8: 0x19acad49 (stale) Mirror 9: 0x19acad49 (stale) Mirror 11: 0x19acad49 (stale) Mirror 13: 0x19acad49 (stale) Mirror 14: 0x19acad49 (stale) Mirror 15: 0x19acad49 (stale) Mirror 16: 0x19acad49 (stale) CRC-32 checksum value for chunk [0, 0x2ed2b): Mirror 1: 0xb4cad35f Mirror 17: 0xb4cad35f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ed2b: skipped 000973 1786827675.825409 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x2ed2b): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ed2b): Mirror 4: 0x19acad49 (stale) Mirror 5: 0x19acad49 (stale) Mirror 6: 0x19acad49 (stale) Mirror 8: 0x19acad49 (stale) Mirror 9: 0x19acad49 (stale) Mirror 11: 0x19acad49 (stale) Mirror 13: 0x19acad49 (stale) Mirror 14: 0x19acad49 (stale) Mirror 15: 0x19acad49 (stale) Mirror 16: 0x19acad49 (stale) CRC-32 checksum value for chunk [0, 0x2ed2b): Mirror 1: 0xb4cad35f Mirror 17: 0xb4cad35f Mirror 18: 0xb4cad35f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ed2b: skipped 000974 1786827677.685086 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x2ed2b): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ed2b): Mirror 4: 0x19acad49 (stale) Mirror 5: 0x19acad49 (stale) Mirror 6: 0x19acad49 (stale) Mirror 8: 0x19acad49 (stale) Mirror 9: 0x19acad49 (stale) Mirror 11: 0x19acad49 (stale) Mirror 13: 0x19acad49 (stale) Mirror 14: 0x19acad49 (stale) Mirror 15: 0x19acad49 (stale) Mirror 16: 0x19acad49 (stale) CRC-32 checksum value for chunk [0, 0x2ed2b): Mirror 1: 0xb4cad35f Mirror 17: 0xb4cad35f Mirror 18: 0xb4cad35f Mirror 19: 0xb4cad35f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ed2b: skipped 000977 1786827679.074815 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x2ed2b): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ed2b): Mirror 4: 0x19acad49 (stale) Mirror 5: 0x19acad49 (stale) Mirror 6: 0x19acad49 (stale) Mirror 8: 0x19acad49 (stale) Mirror 9: 0x19acad49 (stale) Mirror 11: 0x19acad49 (stale) Mirror 13: 0x19acad49 (stale) Mirror 14: 0x19acad49 (stale) Mirror 15: 0x19acad49 (stale) Mirror 16: 0x19acad49 (stale) CRC-32 checksum value for chunk [0, 0x2ed2b): Mirror 1: 0xb4cad35f Mirror 17: 0xb4cad35f Mirror 18: 0xb4cad35f Mirror 19: 0xb4cad35f Mirror 20: 0xb4cad35f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ed2b: skipped 000982 1786827680.411410 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x2ed2b): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ed2b): Mirror 4: 0x19acad49 (stale) Mirror 5: 0x19acad49 (stale) Mirror 6: 0x19acad49 (stale) Mirror 8: 0x19acad49 (stale) Mirror 9: 0x19acad49 (stale) Mirror 11: 0x19acad49 (stale) Mirror 13: 0x19acad49 (stale) Mirror 14: 0x19acad49 (stale) Mirror 15: 0x19acad49 (stale) Mirror 16: 0x19acad49 (stale) CRC-32 checksum value for chunk [0, 0x2ed2b): Mirror 1: 0xb4cad35f Mirror 17: 0xb4cad35f Mirror 18: 0xb4cad35f Mirror 19: 0xb4cad35f Mirror 20: 0xb4cad35f Mirror 21: 0xb4cad35f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ed2b: skipped 000983 1786827681.753425 read 0x01367e thru 0x01ee5e (0xb7e1 bytes) 000986 1786827681.768597 MIRROR_SPLIT mirror 12 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ed2b): Mirror 4: 0x19acad49 (stale) Mirror 5: 0x19acad49 (stale) Mirror 6: 0x19acad49 (stale) Mirror 8: 0x19acad49 (stale) Mirror 9: 0x19acad49 (stale) Mirror 11: 0x19acad49 (stale) Mirror 13: 0x19acad49 (stale) Mirror 14: 0x19acad49 (stale) Mirror 15: 0x19acad49 (stale) Mirror 16: 0x19acad49 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ed2b): Mirror 4: 0x19acad49 (stale) Mirror 5: 0x19acad49 (stale) Mirror 6: 0x19acad49 (stale) Mirror 8: 0x19acad49 (stale) Mirror 9: 0x19acad49 (stale) Mirror 11: 0x19acad49 (stale) Mirror 13: 0x19acad49 (stale) Mirror 14: 0x19acad49 (stale) Mirror 15: 0x19acad49 (stale) Mirror 16: 0x19acad49 (stale) CRC-32 checksum value for chunk [0, 0x2ed2b): Mirror 1: 0xb4cad35f Mirror 17: 0xb4cad35f Mirror 19: 0xb4cad35f Mirror 20: 0xb4cad35f Mirror 21: 0xb4cad35f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ed2b: skipped 000988 1786827682.670226 mapwrite 0x00941f thru 0x0117a7 (0x8389 bytes) 000989 1786827682.945594 mapread 0x015f52 thru 0x021a90 (0xbb3f bytes) 000990 1786827682.966734 trunc from 0x02ed2b to 0x03d1ec (0xe4c2 bytes) 000991 1786827683.092760 MIRROR_SPLIT mirror 5 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d1ed): Mirror 4: 0x84633e4c (stale) Mirror 5: 0x84633e4c (stale) Mirror 6: 0x84633e4c (stale) Mirror 8: 0x84633e4c (stale) Mirror 11: 0x84633e4c (stale) Mirror 13: 0x84633e4c (stale) Mirror 14: 0x84633e4c (stale) Mirror 15: 0x84633e4c (stale) Mirror 16: 0x84633e4c (stale) Mirror 17: 0xd8b8c470 (stale) Mirror 19: 0xd8b8c470 (stale) Mirror 20: 0xd8b8c470 (stale) Mirror 21: 0xd8b8c470 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d1ed): Mirror 4: 0x84633e4c (stale) Mirror 5: 0x84633e4c (stale) Mirror 6: 0x84633e4c (stale) Mirror 8: 0x84633e4c (stale) Mirror 11: 0x84633e4c (stale) Mirror 13: 0x84633e4c (stale) Mirror 14: 0x84633e4c (stale) Mirror 15: 0x84633e4c (stale) Mirror 16: 0x84633e4c (stale) Mirror 17: 0xd8b8c470 (stale) Mirror 19: 0xd8b8c470 (stale) Mirror 20: 0xd8b8c470 (stale) Mirror 21: 0xd8b8c470 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d1ed: skipped 000992 1786827684.006630 mapread 0x01e902 thru 0x0228ba (0x3fb9 bytes) 000993 1786827684.036291 MIRROR_RESYNC 14 mirrors 000994 1786827685.887792 write 0x03621e thru 0x038ef3 (0x2cd6 bytes) 000998 1786827685.987489 trunc from 0x03d1ed to 0x008257 (0x34f95 bytes) 000999 1786827686.013733 trunc from 0x008258 to 0x01d959 (0x15702 bytes) 001000 1786827686.038429 mapwrite 0x029d7a thru 0x030878 (0x6aff bytes) All operations completed A-OK! PASS 70a (107s) == sanity-flr test 71: check mirror extend parent fid ==== 17:01:31 (1786827691) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 17:01:35 (1786827695) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.222908 s, 9.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326532 s, 3.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 17:01:44 (1786827704) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre2 Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3625600 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5622560 .. Extending file size to 4837920 .. Extending file size to 7563904 .. Extending file size to 4866112 .. Extending file size to 8420352 .. Extending file size to 8831392 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8845088 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9031296 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8862752 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57328 57329 57330 57334 57336 done 192.168.203.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre2 (opts:) 192.168.203.126@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (81s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 17:03:05 (1786827785) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre2 Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2592416 .. Extending file size to 2414112 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 4835488 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8238464 .. failed Extending file size to 7948224 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8349920 .. Extending file size to 8374944 .. Extending file size to 8800704 .. Extending file size to 8542720 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8629312 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8884128 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8921824 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9182592 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59096 59097 59100 59101 59102 done 192.168.203.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre2 (opts:) 192.168.203.126@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 17:04:27 (1786827867) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.088329 s, 23.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.311086 s, 6.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x46040fb1 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre2 (opts:) PASS 200c (15s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 17:04:43 (1786827883) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.12251 s, 17.1 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 17:04:49 (1786827889) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fe:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2e7:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 17:04:56 (1786827896) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.159855 s, 13.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x305:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 17:05:02 (1786827902) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.208603 s, 10.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x306:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x307:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ea:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x306:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x307:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ea:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2eb:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 17:05:09 (1786827909) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.230315 s, 9.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x308:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ec:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x308:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ec:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 17:05:41 (1786827941) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg326-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15052 s, 7.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } PASS 204d (58s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 17:06:39 (1786827999) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg326-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.170317 s, 12.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (57s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 17:07:36 (1786828056) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg326-server mds-ost sync done. Creating new pool test_204f oleg326-server: Pool lustre.test_204f created Adding targets to pool oleg326-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg326-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.265792 s, 7.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg326-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg326-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg326-server: Pool lustre.test_204f destroyed PASS 204f (79s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 17:08:55 (1786828135) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 17:09:03 (1786828143) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 17:09:10 (1786828150) oleg326-server: Pool lustre.test_206 created Adding targets to pool oleg326-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg326-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg326-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg326-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg326-server: Pool lustre.test_206 destroyed PASS 206 (24s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 17:09:35 (1786828175) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704228 s, 14.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269321 s, 38.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x574771a9 Mirror 3: 0x574771a9 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 17:09:45 (1786828185) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 17:09:49 (1786828189) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 17:09:52 (1786828192) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.125384 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2190000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x19b0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x19b0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x19b0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2210000 Read this (4 bytes): BBBB 192.168.203.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 17:10:29 (1786828229) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0554184 s, 148 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 17:10:36 (1786828236) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445315 s, 23.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x223 lmm_fid: [0x200000404:0x223:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x223 lmm_fid: [0x200000404:0x223:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834467 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-15 17:10:38.000000000 -0400 Modify: 2026-08-15 17:10:38.000000000 -0400 Change: 2026-08-15 17:10:40.000000000 -0400 Birth: 2026-08-15 17:10:37.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 17:10:46 (1786828246) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 17:10:50 (1786828250) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.787471 s, 52.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0548272 s, 74.7 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 15 17:10 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x225:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 15 17:10 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 17:10:59 (1786828259) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 17:11:07 (1786828267) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1582 sec =========== 17:11:17 (1786828277) === sanity-flr: start cleanup 17:11:19 (1786828279) === === sanity-flr: finish cleanup 17:11:28 (1786828288) ===