-----============= acceptance-small: sanity-flr ============----- Sat Aug 15 20:02:33 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg306-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 20:02:42 (1786838562) === oleg306-client.virtnet: executing check_config_client /mnt/lustre oleg306-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg306-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8e4446351800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8e4446351800.idle_timeout=debug disable quota as required oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 20:02:57 (1786838577) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7625] == sanity-flr test 0a: lfs mirror create with -N option == 20:02:58 (1786838578) PASS 0a (7s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:03:05 (1786838585) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:03:06 (1786838586) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 20:03:07 (1786838587) PASS 0d (10s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:03:17 (1786838597) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:03:18 (1786838598) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 20:03:20 (1786838600) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8e4446351800.stats=clear osc.lustre-OST0001-osc-ffff8e4446351800.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:03:23 (1786838603) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:03:27 (1786838607) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:03:30 (1786838610) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 0m: mirror extend with -N= option ===== 20:03:31 (1786838611) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:03:35 (1786838615) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 20:03:38 (1786838618) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:03:41 (1786838621) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:03:43 (1786838623) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:03:46 (1786838626) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:03:50 (1786838630) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:03:52 (1786838632) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624457 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490505 s, 21.4 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:03:56 (1786838636) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.242533 s, 86.5 MB/s PASS 22 (4s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 20:04:00 (1786838640) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:04:04 (1786838644) Stopping /mnt/lustre-ost1 (opts:) on oleg306-server oleg306-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e4446351800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e4446351800.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg306-server: oleg306-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg306-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg306-server oleg306-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e4446351800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e4446351800.ost_server_uuid in DISCONN state after 2 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg306-server: oleg306-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg306-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (33s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:04:37 (1786838677) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0574404 s, 36.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg306-server oleg306-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e4446351800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e4446351800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg306-server: oleg306-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg306-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (16s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:04:53 (1786838693) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg306-server oleg306-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e4446351800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e4446351800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg306-server: oleg306-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg306-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg306-server oleg306-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e4446351800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e4446351800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg306-server: oleg306-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg306-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (38s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:05:31 (1786838731) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.02577 s, 25.9 MB/s -rw-r--r-- 1 root root 52428800 Aug 15 20:05 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.455s user 0m0.000s sys 0m0.155s Stopping /mnt/lustre-ost1 (opts:) on oleg306-server oleg306-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e4446351800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e4446351800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.433s user 0m0.001s sys 0m0.149s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg306-server: oleg306-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg306-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg306-server oleg306-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e4446351800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e4446351800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.548s user 0m0.001s sys 0m0.174s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg306-server: oleg306-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg306-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg306-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e4446351800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e4446351800.ost_server_uuid in FULL state after 0 sec PASS 33b (42s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:06:13 (1786838773) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:06:15 (1786838775) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:06:16 (1786838776) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 20:06:18 (1786838778) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 20:06:21 (1786838781) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.268684 s, 46.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (5s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:06:26 (1786838786) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120827 s, 86.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184811 s, 56.7 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:06:29 (1786838789) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:06:32 (1786838792) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000753657 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000826023 s, 1.2 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000750871 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000617366 s, 1.7 MB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 20:06:36 (1786838796) create test files with size 15731902 .. 28c2bdfb90bdaa7a9656fa359199dd48 - 4744f41138e55221cdefed2394487947 - 5f24d17a6880360b68bf201d17e63ef4 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg306-client.virtnet /mnt/lustre (opts:) Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre PASS 37 (7s) == sanity-flr test 38: resync ============================ 20:06:43 (1786838803) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (5s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:06:48 (1786838808) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:06:50 (1786838810) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507956 s, 20.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117495 s, 89.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 20:06:53 (1786838813) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0540685 s, 38.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.134869 s, 31.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0724459 s, 28.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (5s) == sanity-flr test 42: lfs mirror verify ================= 20:06:58 (1786838818) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 20:07:00 (1786838820) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:07:01 (1786838821) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239796 s, 43.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 20:07:04 (1786838824) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:07:05 (1786838825) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:07:09 (1786838829) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.118849 s, 88.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg306-client.virtnet /mnt/lustre (opts:) Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (18s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:07:27 (1786838847) llite.lustre-ffff8e44492f5800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.122677 s, 85.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg306-client.virtnet /mnt/lustre (opts:) Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8e4446331800.xattr_cache=1 PASS 44e (6s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:07:33 (1786838853) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (15s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:07:49 (1786838869) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (13s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:08:02 (1786838882) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 20:08:03 (1786838883) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0741111 s, 42.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0643076 s, 48.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x767ee57f (nosync,stale) Mirror 2: 0x767ee57f (nosync,stale) ** verify mirror 2 doesn't change original checksum: 20ffa9dccfc873d9297a2a36c43f0cb1 - mirror 1 checksum: 1adef4209c46085f3c02c6bf5b5c5efc - mirror 2 checksum: 20ffa9dccfc873d9297a2a36c43f0cb1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-15T20:08:04-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-15T20:08:04-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 20ffa9dccfc873d9297a2a36c43f0cb1 - mirror 1 checksum: 1adef4209c46085f3c02c6bf5b5c5efc - mirror 2 checksum: 1adef4209c46085f3c02c6bf5b5c5efc - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-15T20:08:04-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-15T20:08:04-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } PASS 48 (4s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:08:08 (1786838888) Failing mds1 on oleg306-server Stopping /mnt/lustre-mds1 (opts:) on oleg306-server 20:08:10 (1786838890) shut down facet: mds1 facet_host: oleg306-server facet_failover_host: oleg306-server Failover mds1 to oleg306-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-MDT0000 20:08:25 (1786838905) targets are mounted 20:08:25 (1786838905) facet_failover done oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140879 s, 74.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xee:0x0] } PASS 50A (25s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:08:33 (1786838913) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:08:34 (1786838914) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:08:35 (1786838915) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00633723 s, 2.6 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:08:38 (1786838918) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:08:39 (1786838919) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0038806 s, 1.1 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0397911 s, 103 kB/s PASS 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:08:42 (1786838922) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (9s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:08:51 (1786838931) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (21s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:09:12 (1786838952) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:09:30 (1786838970) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:09:48 (1786838988) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 20:09:50 (1786838990) mirror create pid 54239 mirror split pid 54240 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54239 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54240 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 20:10:53 (1786839053) Seed set to 9054 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1786839054.379910 mapwrite 0x030d1c thru 0x038da8 (0x808d bytes) 000002 1786839054.447814 read 0x015f6e thru 0x01e6e5 (0x8778 bytes) 000003 1786839054.457080 MIRROR_EXTEND to 2 mirrors 000007 1786839054.772743 write 0x03910c thru 0x03eb43 (0x5a38 bytes) 000008 1786839054.809560 write 0x0266ef thru 0x02e169 (0x7a7b bytes) truncating to largest ever: 0x2432c 000009 1786839054.815965 trunc from 0x03eb44 to 0x02432b (0x1a818 bytes) 000010 1786839054.845214 write 0x010313 thru 0x01b9bc (0xb6aa bytes) 000012 1786839054.852603 mapwrite 0x01ae2d thru 0x01c666 (0x183a bytes) 000013 1786839054.881474 write 0x00fc63 thru 0x0192f4 (0x9692 bytes) truncating to largest ever: 0x2c15f 000015 1786839054.886156 trunc from 0x02432c to 0x02c15e (0x7e33 bytes) 000017 1786839054.895967 trunc from 0x02c15f to 0x024dff (0x735f bytes) truncating to largest ever: 0x3f425 000018 1786839054.904746 trunc from 0x024e00 to 0x03f424 (0x1a625 bytes) 000019 1786839054.916044 write 0x029e4b thru 0x02a837 (0x9ed bytes) 000020 1786839054.921031 mapwrite 0x0031c0 thru 0x006109 (0x2f4a bytes) 000021 1786839054.952562 mapwrite 0x004e69 thru 0x01377a (0xe912 bytes) 000023 1786839054.988620 read 0x02fb26 thru 0x030efc (0x13d7 bytes) 000024 1786839054.992337 write 0x03e07f thru 0x03ffff (0x1f81 bytes) 000028 1786839054.998002 write 0x031dfd thru 0x03ad5b (0x8f5f bytes) 000029 1786839055.004898 mapwrite 0x02785e thru 0x030558 (0x8cfb bytes) 000030 1786839055.034291 trunc from 0x040000 to 0x03d84b (0x27b4 bytes) 000031 1786839055.048892 mapwrite 0x00207d thru 0x00f266 (0xd1ea bytes) 000034 1786839055.082986 write 0x0077b2 thru 0x015ee8 (0xe737 bytes) 000035 1786839055.084332 write 0x020480 thru 0x02b386 (0xaf07 bytes) 000036 1786839055.085450 read 0x00da16 thru 0x012067 (0x4652 bytes) 000038 1786839055.086398 read 0x002337 thru 0x007660 (0x532a bytes) 000040 1786839055.087393 mapwrite 0x00cc41 thru 0x00f85e (0x2c1e bytes) 000042 1786839055.114037 trunc from 0x03d84c to 0x0074be (0x3638d bytes) 000043 1786839055.123885 mapwrite 0x008688 thru 0x017fcc (0xf945 bytes) 000045 1786839055.163456 write 0x0380c6 thru 0x03a030 (0x1f6b bytes) 000046 1786839055.164442 read 0x0201a8 thru 0x021918 (0x1771 bytes) 000047 1786839055.169999 mapwrite 0x030012 thru 0x03f888 (0xf877 bytes) 000049 1786839055.211624 write 0x006c77 thru 0x0104f5 (0x987f bytes) 000050 1786839055.213230 trunc from 0x03f889 to 0x02dec9 (0x119bf bytes) 000051 1786839055.221990 mapread 0x002417 thru 0x007db5 (0x599f bytes) 000052 1786839055.222997 read 0x018be5 thru 0x01c817 (0x3c33 bytes) 000055 1786839055.224912 trunc from 0x02deca to 0x03d71d (0xf854 bytes) 000057 1786839055.233869 trunc from 0x03d71e to 0x00c4b7 (0x31266 bytes) 000060 1786839055.246527 read 0x009fa4 thru 0x00c4b7 (0x2514 bytes) 000061 1786839055.250458 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) 000062 1786839055.325895 write 0x024ba8 thru 0x02ea95 (0x9eee bytes) 000063 1786839055.327277 trunc from 0x02ea96 to 0x019452 (0x15643 bytes) 000065 1786839055.335907 write 0x02729c thru 0x03448f (0xd1f4 bytes) 000066 1786839055.337332 mapread 0x016570 thru 0x01e5c3 (0x8054 bytes) 000069 1786839055.349609 mapwrite 0x039ece thru 0x03ffff (0x6132 bytes) 000071 1786839055.388547 read 0x0226dc thru 0x02e13e (0xba63 bytes) 000074 1786839055.393794 mapread 0x000664 thru 0x00fdd3 (0xf770 bytes) 000075 1786839055.399544 write 0x03fb82 thru 0x03ffff (0x47e bytes) 000077 1786839055.401157 trunc from 0x040000 to 0x00145d (0x3eba2 bytes) 000079 1786839055.409803 mapread 0x0004fc thru 0x00145d (0xf62 bytes) 000080 1786839055.412697 MIRROR_EXTEND to 2 mirrors 000081 1786839055.711480 write 0x037e8b thru 0x03ffff (0x8175 bytes) 000082 1786839055.747684 trunc from 0x040000 to 0x035dfe (0xa201 bytes) 000083 1786839055.768417 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x35dff): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35dff): Mirror 2: 0x8e95417 (stale) CRC-32 checksum value for chunk [0, 0x35dff): Mirror 1: 0x8e95417 Mirror 3: 0x8e95417 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35dff: skipped 000084 1786839056.080367 trunc from 0x035dff to 0x009970 (0x2c48e bytes) 000085 1786839056.114812 mapread 0x00575c thru 0x009970 (0x4215 bytes) 000091 1786839056.128081 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x9971): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9971): Mirror 2: 0x7e1fbf69 (stale) Mirror 3: 0x7e1fbf69 (stale) CRC-32 checksum value for chunk [0, 0x9971): Mirror 1: 0x7e1fbf69 Mirror 4: 0x7e1fbf69 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9971: skipped 000092 1786839056.439612 trunc from 0x009971 to 0x03de4e (0x344de bytes) 000093 1786839056.478112 read 0x013efd thru 0x022c4f (0xed53 bytes) 000094 1786839056.484643 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3de4f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3de4f): Mirror 2: 0xb7cdd619 (stale) Mirror 3: 0xb7cdd619 (stale) Mirror 4: 0xb7cdd619 (stale) CRC-32 checksum value for chunk [0, 0x3de4f): Mirror 1: 0xb7cdd619 Mirror 5: 0xb7cdd619 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3de4f: skipped 000097 1786839056.876963 trunc from 0x03de4f to 0x02c91a (0x11534 bytes) 000099 1786839056.929412 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c91b): Mirror 3: 0xb345587 (stale) Mirror 4: 0xb345587 (stale) Mirror 5: 0xb345587 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c91b): Mirror 3: 0xb345587 (stale) Mirror 4: 0xb345587 (stale) Mirror 5: 0xb345587 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c91b: skipped 000100 1786839057.084282 mapread 0x01d208 thru 0x021d21 (0x4b1a bytes) 000102 1786839057.094533 read 0x01089c thru 0x011f12 (0x1677 bytes) 000105 1786839057.098850 mapread 0x00999d thru 0x014433 (0xaa97 bytes) 000106 1786839057.106166 write 0x000c98 thru 0x006987 (0x5cf0 bytes) 000107 1786839057.110014 mapwrite 0x01ba19 thru 0x024358 (0x8940 bytes) 000108 1786839057.130154 mapread 0x01a940 thru 0x0290c3 (0xe784 bytes) 000109 1786839057.137005 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c91b): Mirror 3: 0xb345587 (stale) Mirror 5: 0xb345587 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c91b): Mirror 3: 0xb345587 (stale) Mirror 5: 0xb345587 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c91b: skipped 000112 1786839057.275421 mapread 0x01ba0a thru 0x0232ed (0x78e4 bytes) 000114 1786839057.288437 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c91b): Mirror 5: 0xb345587 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c91b): Mirror 5: 0xb345587 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c91b: skipped 000115 1786839057.394491 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2c91b): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c91b): Mirror 5: 0xb345587 (stale) CRC-32 checksum value for chunk [0, 0x2c91b): Mirror 1: 0xe9fa9212 Mirror 6: 0xe9fa9212 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c91b: skipped 000116 1786839057.817353 read 0x0191c0 thru 0x01a7d0 (0x1611 bytes) 000117 1786839057.822720 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2c91b): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c91b): Mirror 5: 0xb345587 (stale) CRC-32 checksum value for chunk [0, 0x2c91b): Mirror 1: 0xe9fa9212 Mirror 6: 0xe9fa9212 Mirror 7: 0xe9fa9212 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c91b: skipped 000118 1786839058.231267 write 0x0332ba thru 0x03ffff (0xcd46 bytes) 000119 1786839058.264978 trunc from 0x040000 to 0x0283ac (0x17c53 bytes) 000123 1786839058.277745 read 0x0033d9 thru 0x005315 (0x1f3d bytes) 000125 1786839058.281973 mapread 0x0197b9 thru 0x01d90d (0x4155 bytes) 000126 1786839058.298685 mapread 0x007831 thru 0x0164a0 (0xec70 bytes) 000127 1786839058.310054 write 0x037d2f thru 0x03eda2 (0x7074 bytes) 000128 1786839058.311539 write 0x02f027 thru 0x031fa0 (0x2f7a bytes) 000129 1786839058.317545 read 0x00b751 thru 0x016ae2 (0xb392 bytes) 000131 1786839058.318557 read 0x008ca5 thru 0x00e7e3 (0x5b3f bytes) 000132 1786839058.319128 mapwrite 0x00f2ce thru 0x01af64 (0xbc97 bytes) 000133 1786839058.349481 mapwrite 0x00421a thru 0x00fbb9 (0xb9a0 bytes) 000134 1786839058.372140 read 0x03b8dd thru 0x03eda2 (0x34c6 bytes) 000135 1786839058.372687 mapread 0x011703 thru 0x0204a3 (0xeda1 bytes) 000137 1786839058.374190 mapwrite 0x0159b0 thru 0x024509 (0xeb5a bytes) 000139 1786839058.403546 trunc from 0x03eda3 to 0x016629 (0x28779 bytes) 000141 1786839058.414466 trunc from 0x01662a to 0x03ce19 (0x267f0 bytes) 000144 1786839058.423995 write 0x03ed36 thru 0x03ffff (0x12ca bytes) 000147 1786839058.425897 write 0x02c121 thru 0x038b8f (0xca6f bytes) 000148 1786839058.431541 mapwrite 0x028388 thru 0x037c22 (0xf89b bytes) 000149 1786839058.467429 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe20eea22 (stale) Mirror 6: 0x55e6c896 (stale) Mirror 7: 0x55e6c896 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x919d00af Mirror 8: 0x919d00af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000153 1786839058.896296 read 0x03bf1e thru 0x03ffff (0x40e2 bytes) 000154 1786839058.905527 trunc from 0x040000 to 0x00c620 (0x339df bytes) 000155 1786839058.957547 trunc from 0x00c621 to 0x03f001 (0x329e1 bytes) 000156 1786839058.970700 MIRROR_RESYNC 5 mirrors 000157 1786839059.421145 mapwrite 0x033694 thru 0x03a990 (0x72fd bytes) 000162 1786839059.494312 mapwrite 0x02c38a thru 0x034fdb (0x8c52 bytes) 000163 1786839059.523113 read 0x020c14 thru 0x02dd6a (0xd157 bytes) 000165 1786839059.528477 mapwrite 0x0257fd thru 0x02b5da (0x5dde bytes) 000167 1786839059.556204 MIRROR_RESYNC 5 mirrors 000168 1786839060.187801 read 0x03254f thru 0x03e823 (0xc2d5 bytes) 000170 1786839060.192762 trunc from 0x03f002 to 0x00a376 (0x34c8b bytes) 000171 1786839060.228352 read 0x007f02 thru 0x00a376 (0x2475 bytes) 000174 1786839060.234134 read 0x001614 thru 0x008775 (0x7162 bytes) 000175 1786839060.234808 mapread 0x003659 thru 0x00730e (0x3cb6 bytes) 000176 1786839060.235714 trunc from 0x00a377 to 0x010b33 (0x67bd bytes) 000178 1786839060.245749 read 0x006901 thru 0x010b33 (0xa233 bytes) 000183 1786839060.252384 mapwrite 0x007e09 thru 0x00e2b7 (0x64af bytes) 000185 1786839060.278297 write 0x026788 thru 0x0321db (0xba54 bytes) 000186 1786839060.279729 mapread 0x0011b3 thru 0x006ef7 (0x5d45 bytes) 000187 1786839060.280734 trunc from 0x0321dc to 0x03883d (0x6662 bytes) 000189 1786839060.290710 trunc from 0x03883e to 0x00f92f (0x28f0e bytes) 000191 1786839060.304247 write 0x0335a4 thru 0x035b37 (0x2594 bytes) 000192 1786839060.308748 MIRROR_RESYNC 5 mirrors 000193 1786839060.852309 write 0x032739 thru 0x03379a (0x1062 bytes) 000196 1786839060.882505 trunc from 0x035b38 to 0x009de9 (0x2bd4e bytes) 000197 1786839060.897335 trunc from 0x009dea to 0x002ff6 (0x6df3 bytes) 000198 1786839060.904275 write 0x008f87 thru 0x00ba4d (0x2ac7 bytes) 000199 1786839060.907980 MIRROR_RESYNC 5 mirrors 000200 1786839061.447784 write 0x00adc5 thru 0x01301e (0x825a bytes) 000203 1786839061.471536 read 0x00e517 thru 0x012c79 (0x4763 bytes) 000204 1786839061.472181 write 0x03cb0f thru 0x03ffff (0x34f1 bytes) 000206 1786839061.473418 write 0x01956d thru 0x027960 (0xe3f4 bytes) 000207 1786839061.477554 write 0x01a1d8 thru 0x023618 (0x9441 bytes) 000209 1786839061.479161 write 0x025111 thru 0x02b4ab (0x639b bytes) 000210 1786839061.482758 mapread 0x00c356 thru 0x0155a8 (0x9253 bytes) 000212 1786839061.490454 trunc from 0x040000 to 0x02d49b (0x12b64 bytes) 000214 1786839061.501212 read 0x008c53 thru 0x00cf69 (0x4317 bytes) 000215 1786839061.507553 trunc from 0x02d49c to 0x03c419 (0xef7e bytes) 000217 1786839061.517027 write 0x006ddd thru 0x00c0bc (0x52e0 bytes) 000218 1786839061.518090 mapwrite 0x01a101 thru 0x021fbf (0x7ebf bytes) 000220 1786839061.549856 trunc from 0x03c41a to 0x03b5f3 (0xe26 bytes) 000221 1786839061.559862 mapread 0x024e52 thru 0x02e5a2 (0x9751 bytes) 000222 1786839061.565193 read 0x017bb3 thru 0x01a14c (0x259a bytes) 000223 1786839061.565770 read 0x00421a thru 0x00894d (0x4734 bytes) 000225 1786839061.569731 MIRROR_RESYNC 5 mirrors 000226 1786839062.286318 read 0x02a227 thru 0x033e52 (0x9c2c bytes) 000230 1786839062.291062 trunc from 0x03b5f4 to 0x010b0c (0x2aae7 bytes) 000231 1786839062.318658 write 0x006b10 thru 0x00d9f8 (0x6ee9 bytes) 000232 1786839062.323600 mapwrite 0x0260fb thru 0x03527f (0xf185 bytes) 000233 1786839062.377372 write 0x03fbf9 thru 0x03ffff (0x407 bytes) 000234 1786839062.382423 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8cc6ba8c (stale) Mirror 7: 0x8cc6ba8c (stale) Mirror 8: 0x8cc6ba8c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8cc6ba8c (stale) Mirror 7: 0x8cc6ba8c (stale) Mirror 8: 0x8cc6ba8c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000235 1786839062.556801 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8cc6ba8c (stale) Mirror 7: 0x8cc6ba8c (stale) Mirror 8: 0x8cc6ba8c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xaf2e4042 Mirror 9: 0xaf2e4042 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000236 1786839063.015937 mapread 0x004d99 thru 0x01224d (0xd4b5 bytes) 000237 1786839063.036519 write 0x039de8 thru 0x03e38a (0x45a3 bytes) 000239 1786839063.071564 read 0x037d71 thru 0x03ffff (0x828f bytes) 000242 1786839063.077055 trunc from 0x040000 to 0x0137e8 (0x2c817 bytes) 000245 1786839063.087497 mapread 0x0066c4 thru 0x0137e8 (0xd125 bytes) 000247 1786839063.097110 MIRROR_RESYNC 5 mirrors 000248 1786839063.592218 write 0x03b995 thru 0x03ffff (0x466b bytes) 000249 1786839063.619551 read 0x0383a6 thru 0x03ffff (0x7c5a bytes) 000250 1786839063.622099 read 0x038a7a thru 0x03ffff (0x7586 bytes) 000251 1786839063.622639 read 0x032dfd thru 0x03432f (0x1533 bytes) 000253 1786839063.630938 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xa9ffa78 (stale) Mirror 7: 0xa9ffa78 (stale) Mirror 8: 0xa9ffa78 (stale) Mirror 9: 0xa9ffa78 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4c5229a9 Mirror 10: 0x4c5229a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000255 1786839064.058069 trunc from 0x040000 to 0x001ab7 (0x3e548 bytes) 000258 1786839064.101355 mapwrite 0x00072f thru 0x0022c0 (0x1b92 bytes) 000260 1786839064.145645 write 0x024b47 thru 0x030eff (0xc3b9 bytes) 000261 1786839064.149393 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30f00): Mirror 6: 0x76f919dc (stale) Mirror 7: 0x76f919dc (stale) Mirror 8: 0x76f919dc (stale) Mirror 10: 0x76f919dc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30f00): Mirror 6: 0x76f919dc (stale) Mirror 7: 0x76f919dc (stale) Mirror 8: 0x76f919dc (stale) Mirror 10: 0x76f919dc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30f00: skipped 000262 1786839064.332993 mapwrite 0x01ea4a thru 0x02d099 (0xe650 bytes) 000263 1786839064.400258 mapread 0x0280bc thru 0x02e783 (0x66c8 bytes) 000264 1786839064.401017 read 0x01e9ce thru 0x02829b (0x98ce bytes) 000265 1786839064.401669 mapwrite 0x024867 thru 0x0288ff (0x4099 bytes) 000266 1786839064.432722 write 0x011014 thru 0x01af34 (0x9f21 bytes) 000267 1786839064.436905 write 0x009a1b thru 0x016ad9 (0xd0bf bytes) 000268 1786839064.441631 read 0x006b7d thru 0x009828 (0x2cac bytes) 000270 1786839064.449249 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30f00): Mirror 7: 0x76f919dc (stale) Mirror 8: 0x76f919dc (stale) Mirror 10: 0x76f919dc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30f00): Mirror 7: 0x76f919dc (stale) Mirror 8: 0x76f919dc (stale) Mirror 10: 0x76f919dc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30f00: skipped 000271 1786839064.595809 write 0x0032ff thru 0x00e13b (0xae3d bytes) 000272 1786839064.601704 mapread 0x004e29 thru 0x0097ff (0x49d7 bytes) 000274 1786839064.603187 mapread 0x015202 thru 0x015f79 (0xd78 bytes) 000276 1786839064.607157 mapwrite 0x027739 thru 0x02b5f3 (0x3ebb bytes) 000277 1786839064.640655 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x30f00): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30f00): Mirror 7: 0x76f919dc (stale) Mirror 8: 0x76f919dc (stale) Mirror 10: 0x76f919dc (stale) CRC-32 checksum value for chunk [0, 0x30f00): Mirror 1: 0xcb053b19 Mirror 11: 0xcb053b19 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30f00: skipped 000278 1786839065.009132 mapwrite 0x015ff9 thru 0x019198 (0x31a0 bytes) 000279 1786839065.104923 read 0x0154c8 thru 0x0195f3 (0x412c bytes) 000283 1786839065.110325 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x30f00): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30f00): Mirror 7: 0x76f919dc (stale) Mirror 8: 0x76f919dc (stale) Mirror 10: 0x76f919dc (stale) Mirror 11: 0xcb053b19 (stale) CRC-32 checksum value for chunk [0, 0x30f00): Mirror 1: 0x3a732ac8 Mirror 12: 0x3a732ac8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30f00: skipped 000285 1786839065.506414 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30f00): Mirror 7: 0x76f919dc (stale) Mirror 8: 0x76f919dc (stale) Mirror 10: 0x76f919dc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30f00): Mirror 7: 0x76f919dc (stale) Mirror 8: 0x76f919dc (stale) Mirror 10: 0x76f919dc (stale) CRC-32 checksum value for chunk [0, 0x30f00): Mirror 1: 0x3a732ac8 Mirror 12: 0x3a732ac8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30f00: skipped 000286 1786839065.811436 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30f00): Mirror 7: 0x76f919dc (stale) Mirror 8: 0x76f919dc (stale) Mirror 10: 0x76f919dc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30f00): Mirror 7: 0x76f919dc (stale) Mirror 8: 0x76f919dc (stale) Mirror 10: 0x76f919dc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30f00: skipped 000287 1786839065.963055 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x30f00): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30f00): Mirror 7: 0x76f919dc (stale) Mirror 8: 0x76f919dc (stale) Mirror 10: 0x76f919dc (stale) CRC-32 checksum value for chunk [0, 0x30f00): Mirror 1: 0x3a732ac8 Mirror 11: 0x3a732ac8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30f00: skipped 000289 1786839066.357176 read 0x01ae55 thru 0x02135a (0x6506 bytes) 000291 1786839066.370605 trunc from 0x030f00 to 0x03992e (0x8a2f bytes) 000293 1786839066.407188 mapwrite 0x00fda8 thru 0x01f427 (0xf680 bytes) 000300 1786839066.449082 mapread 0x026e17 thru 0x0287d5 (0x19bf bytes) 000302 1786839066.453901 mapwrite 0x025284 thru 0x030de3 (0xbb60 bytes) 000304 1786839066.474750 write 0x0225bd thru 0x029e93 (0x78d7 bytes) 000310 1786839066.478251 read 0x03470a thru 0x0350d9 (0x9d0 bytes) 000311 1786839066.478814 mapwrite 0x018a0a thru 0x01e1b4 (0x57ab bytes) 000312 1786839066.499718 mapwrite 0x038cef thru 0x03ffff (0x7311 bytes) 000313 1786839066.541242 trunc from 0x040000 to 0x001034 (0x3efcb bytes) 000316 1786839066.552207 write 0x003e61 thru 0x00bee9 (0x8089 bytes) 000317 1786839066.555930 MIRROR_RESYNC 5 mirrors 000320 1786839067.015885 trunc from 0x00beea to 0x03964f (0x2d766 bytes) 000322 1786839067.047933 mapread 0x0061e0 thru 0x01398d (0xd7ae bytes) 000323 1786839067.062274 mapread 0x017105 thru 0x0264b4 (0xf3b0 bytes) 000324 1786839067.071231 write 0x00d3d6 thru 0x015026 (0x7c51 bytes) 000325 1786839067.072536 read 0x02cee8 thru 0x032b2a (0x5c43 bytes) 000327 1786839067.080415 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x39650): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39650): Mirror 7: 0x9a8195a7 (stale) Mirror 8: 0x9a8195a7 (stale) Mirror 10: 0x9a8195a7 (stale) Mirror 11: 0x9a8195a7 (stale) CRC-32 checksum value for chunk [0, 0x39650): Mirror 1: 0x5a3544 Mirror 12: 0x5a3544 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39650: skipped 000330 1786839067.503133 read 0x022cc2 thru 0x0248fd (0x1c3c bytes) 000331 1786839067.506614 mapwrite 0x038cea thru 0x03f9fb (0x6d12 bytes) 000333 1786839067.586504 write 0x01746b thru 0x01ac91 (0x3827 bytes) 000334 1786839067.594387 MIRROR_RESYNC 6 mirrors 000335 1786839068.566169 write 0x01e166 thru 0x02cd7f (0xec1a bytes) 000336 1786839068.593179 read 0x03a538 thru 0x03f9fb (0x54c4 bytes) 000339 1786839068.599683 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f9fc): Mirror 7: 0xde435ee5 (stale) Mirror 8: 0xde435ee5 (stale) Mirror 11: 0xde435ee5 (stale) Mirror 12: 0xde435ee5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f9fc): Mirror 7: 0xde435ee5 (stale) Mirror 8: 0xde435ee5 (stale) Mirror 11: 0xde435ee5 (stale) Mirror 12: 0xde435ee5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f9fc: skipped 000340 1786839068.742896 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3f9fc): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f9fc): Mirror 7: 0xde435ee5 (stale) Mirror 8: 0xde435ee5 (stale) Mirror 11: 0xde435ee5 (stale) Mirror 12: 0xde435ee5 (stale) CRC-32 checksum value for chunk [0, 0x3f9fc): Mirror 1: 0x66293b08 Mirror 13: 0x66293b08 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f9fc: skipped 000341 1786839069.151631 mapread 0x034096 thru 0x035fbf (0x1f2a bytes) 000342 1786839069.157398 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3f9fc): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f9fc): Mirror 7: 0xde435ee5 (stale) Mirror 8: 0xde435ee5 (stale) Mirror 11: 0xde435ee5 (stale) Mirror 12: 0xde435ee5 (stale) CRC-32 checksum value for chunk [0, 0x3f9fc): Mirror 1: 0x66293b08 Mirror 13: 0x66293b08 Mirror 14: 0x66293b08 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f9fc: skipped 000343 1786839069.591185 write 0x03a9cc thru 0x03e704 (0x3d39 bytes) 000344 1786839069.630941 mapwrite 0x035e69 thru 0x03f0c2 (0x925a bytes) 000345 1786839069.670869 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f9fc): Mirror 8: 0xde435ee5 (stale) Mirror 11: 0xde435ee5 (stale) Mirror 12: 0xde435ee5 (stale) Mirror 13: 0x66293b08 (stale) Mirror 14: 0x66293b08 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f9fc): Mirror 8: 0xde435ee5 (stale) Mirror 11: 0xde435ee5 (stale) Mirror 12: 0xde435ee5 (stale) Mirror 13: 0x66293b08 (stale) Mirror 14: 0x66293b08 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f9fc: skipped 000346 1786839069.830048 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3f9fc): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f9fc): Mirror 8: 0xde435ee5 (stale) Mirror 11: 0xde435ee5 (stale) Mirror 12: 0xde435ee5 (stale) Mirror 13: 0x66293b08 (stale) Mirror 14: 0x66293b08 (stale) CRC-32 checksum value for chunk [0, 0x3f9fc): Mirror 1: 0x32f83255 Mirror 15: 0x32f83255 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f9fc: skipped 000347 1786839070.363100 trunc from 0x03f9fc to 0x02a8e4 (0x15117 bytes) 000348 1786839070.392290 mapread 0x01fd77 thru 0x02a8e4 (0xab6e bytes) 000349 1786839070.403332 read 0x00cd58 thru 0x00d3e5 (0x68e bytes) 000350 1786839070.407329 mapread 0x00bc39 thru 0x01a3b1 (0xe779 bytes) 000352 1786839070.414786 write 0x030b52 thru 0x038fd6 (0x8485 bytes) 000354 1786839070.416443 mapwrite 0x01be4d thru 0x02421b (0x83cf bytes) 000356 1786839070.443658 MIRROR_RESYNC 7 mirrors 000357 1786839071.390655 mapwrite 0x002c66 thru 0x00830b (0x56a6 bytes) 000358 1786839071.454193 read 0x00ed8b thru 0x01defd (0xf173 bytes) 000359 1786839071.458904 mapread 0x009457 thru 0x00f607 (0x61b1 bytes) 000360 1786839071.459877 mapread 0x038577 thru 0x038fd6 (0xa60 bytes) 000361 1786839071.466610 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x38fd7): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38fd7): Mirror 8: 0xa2753967 (stale) Mirror 11: 0xa2753967 (stale) Mirror 12: 0xa2753967 (stale) Mirror 13: 0xa2753967 (stale) Mirror 14: 0xa2753967 (stale) Mirror 15: 0xa2753967 (stale) CRC-32 checksum value for chunk [0, 0x38fd7): Mirror 1: 0xd86978c0 Mirror 16: 0xd86978c0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38fd7: skipped 000362 1786839071.806883 trunc from 0x038fd7 to 0x03c542 (0x356c bytes) 000363 1786839071.834187 mapread 0x004892 thru 0x009a91 (0x5200 bytes) 000366 1786839071.841179 read 0x036b9b thru 0x03c3f2 (0x5858 bytes) 000369 1786839071.845155 mapwrite 0x017704 thru 0x020a44 (0x9341 bytes) 000370 1786839071.865698 mapwrite 0x01a66e thru 0x029b72 (0xf505 bytes) 000371 1786839071.887858 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3c543): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c543): Mirror 8: 0x2428462f (stale) Mirror 11: 0x2428462f (stale) Mirror 12: 0x2428462f (stale) Mirror 13: 0x2428462f (stale) Mirror 14: 0x2428462f (stale) Mirror 15: 0x2428462f (stale) Mirror 16: 0x229f11e6 (stale) CRC-32 checksum value for chunk [0, 0x3c543): Mirror 1: 0x8d7dded8 Mirror 17: 0x8d7dded8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c543: skipped 000374 1786839072.276803 mapread 0x037a88 thru 0x03c542 (0x4abb bytes) 000375 1786839072.283721 read 0x031164 thru 0x037344 (0x61e1 bytes) 000376 1786839072.285473 read 0x00b5f2 thru 0x019201 (0xdc10 bytes) 000377 1786839072.289659 read 0x010206 thru 0x019a9f (0x989a bytes) 000378 1786839072.290077 mapwrite 0x00f64f thru 0x01d90f (0xe2c1 bytes) 000381 1786839072.354447 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3c543): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c543): Mirror 8: 0x2428462f (stale) Mirror 11: 0x2428462f (stale) Mirror 12: 0x2428462f (stale) Mirror 13: 0x2428462f (stale) Mirror 14: 0x2428462f (stale) Mirror 15: 0x2428462f (stale) Mirror 16: 0x229f11e6 (stale) Mirror 17: 0x8d7dded8 (stale) CRC-32 checksum value for chunk [0, 0x3c543): Mirror 1: 0xdff62d3 Mirror 18: 0xdff62d3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c543: skipped 000386 1786839072.789395 write 0x032fa1 thru 0x03f14f (0xc1af bytes) 000387 1786839072.821301 mapwrite 0x019276 thru 0x01a8e8 (0x1673 bytes) 000392 1786839072.846773 mapread 0x0016cc thru 0x00a806 (0x913b bytes) 000393 1786839072.851309 read 0x0346ea thru 0x03f14f (0xaa66 bytes) 000396 1786839072.853161 mapwrite 0x01a805 thru 0x021c7e (0x747a bytes) 000398 1786839072.879965 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3f150): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f150): Mirror 8: 0xb340c324 (stale) Mirror 11: 0xb340c324 (stale) Mirror 12: 0xb340c324 (stale) Mirror 13: 0xb340c324 (stale) Mirror 14: 0xb340c324 (stale) Mirror 15: 0xb340c324 (stale) Mirror 16: 0x2bbc37ce (stale) Mirror 17: 0xdb887338 (stale) Mirror 18: 0xa1387f35 (stale) CRC-32 checksum value for chunk [0, 0x3f150): Mirror 1: 0xcd273ab0 Mirror 19: 0xcd273ab0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f150: skipped 000401 1786839073.348947 mapread 0x036140 thru 0x03638d (0x24e bytes) 000402 1786839073.359771 trunc from 0x03f150 to 0x01e136 (0x21019 bytes) 000403 1786839073.393190 mapread 0x019270 thru 0x01df3f (0x4cd0 bytes) 000404 1786839073.401184 mapread 0x01327a thru 0x01e136 (0xaebd bytes) 000406 1786839073.402901 read 0x00af09 thru 0x016832 (0xb92a bytes) 000408 1786839073.407623 trunc from 0x01e137 to 0x02ad12 (0xcbdc bytes) 000409 1786839073.416987 mapwrite 0x03563a thru 0x03bd23 (0x66ea bytes) 000410 1786839073.454415 read 0x033375 thru 0x039b31 (0x67bd bytes) 000411 1786839073.457492 MIRROR_RESYNC 11 mirrors 000412 1786839074.355511 write 0x03c4af thru 0x03f813 (0x3365 bytes) 000415 1786839074.373237 read 0x00ebc3 thru 0x0188d9 (0x9d17 bytes) 000416 1786839074.376687 trunc from 0x03f814 to 0x029a54 (0x15dbf bytes) 000418 1786839074.384119 mapread 0x00281d thru 0x007f98 (0x577c bytes) 000419 1786839074.391343 trunc from 0x029a55 to 0x02f7c8 (0x5d74 bytes) 000421 1786839074.397352 write 0x01331a thru 0x017d5e (0x4a45 bytes) 000423 1786839074.398664 write 0x03e010 thru 0x03ffff (0x1ff0 bytes) 000426 1786839074.400227 trunc from 0x040000 to 0x03633c (0x9cc3 bytes) 000428 1786839074.405811 trunc from 0x03633d to 0x03a09d (0x3d61 bytes) 000429 1786839074.410485 trunc from 0x03a09e to 0x015e50 (0x2424d bytes) 000430 1786839074.415458 mapwrite 0x039e65 thru 0x03ecb0 (0x4e4c bytes) 000431 1786839074.444355 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3ecb1): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ecb1): Mirror 8: 0x1c230c31 (stale) Mirror 11: 0x1c230c31 (stale) Mirror 12: 0x1c230c31 (stale) Mirror 13: 0x1c230c31 (stale) Mirror 14: 0x1c230c31 (stale) Mirror 15: 0x1c230c31 (stale) Mirror 16: 0x1c230c31 (stale) Mirror 17: 0x1c230c31 (stale) Mirror 18: 0x1c230c31 (stale) Mirror 19: 0x1c230c31 (stale) CRC-32 checksum value for chunk [0, 0x3ecb1): Mirror 1: 0xeed5ad44 Mirror 20: 0xeed5ad44 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ecb1: skipped 000432 1786839074.800755 mapread 0x02b269 thru 0x034eb9 (0x9c51 bytes) 000436 1786839074.817294 mapwrite 0x01ea20 thru 0x020483 (0x1a64 bytes) 000437 1786839074.881832 trunc from 0x03ecb1 to 0x02f8d7 (0xf3d9 bytes) 000438 1786839074.894004 mapwrite 0x00fb54 thru 0x013612 (0x3abf bytes) 000439 1786839074.914215 MIRROR_RESYNC 12 mirrors 000440 1786839075.968400 MIRROR_RESYNC 12 mirrors truncating to largest ever: 0x3fd51 000441 1786839076.188006 trunc from 0x02f8d8 to 0x03fd50 (0x10479 bytes) 000442 1786839076.210304 write 0x02669a thru 0x02a85a (0x41c1 bytes) 000443 1786839076.217132 MIRROR_RESYNC 12 mirrors 000444 1786839077.629174 write 0x002028 thru 0x0035b3 (0x158c bytes) 000445 1786839077.650885 trunc from 0x03fd51 to 0x0155e9 (0x2a767 bytes) 000447 1786839077.659139 mapwrite 0x011bbd thru 0x01a03f (0x8483 bytes) 000449 1786839077.694274 mapwrite 0x024133 thru 0x024554 (0x422 bytes) 000450 1786839077.719302 trunc from 0x024555 to 0x023778 (0xddc bytes) 000451 1786839077.725228 trunc from 0x023779 to 0x00eff4 (0x14784 bytes) 000454 1786839077.731413 read 0x008f0c thru 0x00eff4 (0x60e9 bytes) 000455 1786839077.733838 mapwrite 0x0185bf thru 0x02557f (0xcfc1 bytes) 000456 1786839077.762436 read 0x020bad thru 0x02557f (0x49d3 bytes) 000458 1786839077.763655 read 0x01c2cb thru 0x02557f (0x92b5 bytes) 000459 1786839077.764271 write 0x01634e thru 0x0256f8 (0xf3ab bytes) 000460 1786839077.765536 trunc from 0x0256f9 to 0x030877 (0xb17f bytes) 000461 1786839077.772062 mapwrite 0x038ace thru 0x03ffff (0x7532 bytes) 000462 1786839077.801100 mapread 0x03a72d thru 0x03ffff (0x58d3 bytes) 000463 1786839077.802019 mapwrite 0x016046 thru 0x016825 (0x7e0 bytes) 000464 1786839077.819693 MIRROR_RESYNC 12 mirrors 000465 1786839079.006404 mapwrite 0x012ec2 thru 0x0150bc (0x21fb bytes) 000466 1786839079.060036 read 0x03e0a7 thru 0x03ffff (0x1f59 bytes) 000471 1786839079.064145 write 0x00553a thru 0x0069d9 (0x14a0 bytes) 000472 1786839079.069022 MIRROR_RESYNC 12 mirrors 000473 1786839080.282180 write 0x028116 thru 0x031002 (0x8eed bytes) 000474 1786839080.307822 trunc from 0x040000 to 0x033819 (0xc7e6 bytes) 000475 1786839080.315427 mapwrite 0x02e243 thru 0x030965 (0x2723 bytes) 000477 1786839080.333668 write 0x0370b8 thru 0x03ffff (0x8f48 bytes) 000479 1786839080.335499 write 0x02a340 thru 0x038cc2 (0xe983 bytes) 000482 1786839080.337760 write 0x017d78 thru 0x01d81c (0x5aa5 bytes) 000485 1786839080.344477 MIRROR_SPLIT mirror 5 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe9b3616d (stale) Mirror 11: 0xe9b3616d (stale) Mirror 12: 0xe9b3616d (stale) Mirror 13: 0xe9b3616d (stale) Mirror 15: 0xe9b3616d (stale) Mirror 16: 0xe9b3616d (stale) Mirror 17: 0xe9b3616d (stale) Mirror 18: 0xe9b3616d (stale) Mirror 19: 0xe9b3616d (stale) Mirror 20: 0xe9b3616d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe9b3616d (stale) Mirror 11: 0xe9b3616d (stale) Mirror 12: 0xe9b3616d (stale) Mirror 13: 0xe9b3616d (stale) Mirror 15: 0xe9b3616d (stale) Mirror 16: 0xe9b3616d (stale) Mirror 17: 0xe9b3616d (stale) Mirror 18: 0xe9b3616d (stale) Mirror 19: 0xe9b3616d (stale) Mirror 20: 0xe9b3616d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000486 1786839080.540976 read 0x03091c thru 0x03bd0f (0xb3f4 bytes) 000487 1786839080.544838 mapread 0x025971 thru 0x031b61 (0xc1f1 bytes) 000491 1786839080.553650 mapwrite 0x00b277 thru 0x019dda (0xeb64 bytes) 000492 1786839080.581028 read 0x02e8ec thru 0x031e03 (0x3518 bytes) 000494 1786839080.582288 write 0x03d7b6 thru 0x03f035 (0x1880 bytes) 000495 1786839080.586433 write 0x038943 thru 0x03cf3a (0x45f8 bytes) 000496 1786839080.587493 trunc from 0x040000 to 0x030bf9 (0xf406 bytes) 000498 1786839080.595521 mapread 0x014130 thru 0x0217e5 (0xd6b6 bytes) 000499 1786839080.599405 MIRROR_RESYNC 11 mirrors 000500 1786839081.333214 write 0x01635c thru 0x01d81d (0x74c2 bytes) 000502 1786839081.365943 mapwrite 0x038b8d thru 0x03ffff (0x7473 bytes) 000504 1786839081.411219 trunc from 0x040000 to 0x022cca (0x1d335 bytes) 000505 1786839081.419366 mapwrite 0x0232de thru 0x029806 (0x6529 bytes) 000507 1786839081.448670 read 0x00184b thru 0x00c6e7 (0xae9d bytes) 000509 1786839081.452693 mapread 0x01483a thru 0x01dab7 (0x927e bytes) 000510 1786839081.457122 trunc from 0x029807 to 0x00b1b4 (0x1e652 bytes) 000511 1786839081.463689 read 0x001969 thru 0x0019d5 (0x6d bytes) 000513 1786839081.464719 read 0x0009f2 thru 0x001f8b (0x159a bytes) 000514 1786839081.467496 mapread 0x008264 thru 0x00b1b4 (0x2f51 bytes) 000515 1786839081.468411 read 0x000368 thru 0x00af75 (0xac0e bytes) 000516 1786839081.470862 MIRROR_RESYNC 11 mirrors 000517 1786839081.974354 write 0x02038b thru 0x02fd3a (0xf9b0 bytes) 000518 1786839082.006878 mapread 0x00f7d6 thru 0x013551 (0x3d7c bytes) 000519 1786839082.014956 mapwrite 0x008e33 thru 0x0153ee (0xc5bc bytes) 000520 1786839082.044666 mapread 0x010df7 thru 0x01bcd4 (0xaede bytes) 000523 1786839082.050068 mapwrite 0x02194e thru 0x02907c (0x772f bytes) 000524 1786839082.065961 mapwrite 0x0097a3 thru 0x00cd98 (0x35f6 bytes) 000525 1786839082.079959 trunc from 0x02fd3b to 0x0207ab (0xf58f bytes) 000526 1786839082.093033 mapwrite 0x028c7a thru 0x036dda (0xe161 bytes) 000528 1786839082.123149 mapwrite 0x03bd13 thru 0x03ffff (0x42ed bytes) 000529 1786839082.146702 read 0x02cd4e thru 0x031622 (0x48d5 bytes) 000530 1786839082.147301 trunc from 0x040000 to 0x012aa5 (0x2d55a bytes) 000532 1786839082.155261 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x12aa6): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12aa6): Mirror 8: 0x5fc1b37f (stale) Mirror 11: 0x5fc1b37f (stale) Mirror 12: 0x5fc1b37f (stale) Mirror 13: 0x5fc1b37f (stale) Mirror 15: 0x5fc1b37f (stale) Mirror 16: 0x5fc1b37f (stale) Mirror 17: 0x5fc1b37f (stale) Mirror 18: 0x5fc1b37f (stale) Mirror 19: 0x5fc1b37f (stale) Mirror 20: 0x5fc1b37f (stale) CRC-32 checksum value for chunk [0, 0x12aa6): Mirror 1: 0x53424c8 Mirror 21: 0x53424c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12aa6: skipped 000533 1786839082.491544 read 0x00bc20 thru 0x012aa5 (0x6e86 bytes) 000534 1786839082.498642 mapread 0x00c4e0 thru 0x012aa5 (0x65c6 bytes) 000535 1786839082.506597 write 0x025737 thru 0x0263d2 (0xc9c bytes) 000536 1786839082.534999 mapwrite 0x025881 thru 0x02c6f1 (0x6e71 bytes) 000538 1786839082.571693 mapread 0x00745d thru 0x010264 (0x8e08 bytes) 000539 1786839082.576613 read 0x0103e8 thru 0x0179a7 (0x75c0 bytes) 000541 1786839082.581661 mapwrite 0x0123dc thru 0x015335 (0x2f5a bytes) 000543 1786839082.600672 read 0x025e77 thru 0x02c6f1 (0x687b bytes) 000545 1786839082.601769 trunc from 0x02c6f2 to 0x01b16e (0x11583 bytes) 000546 1786839082.607433 mapread 0x0066c1 thru 0x00b478 (0x4db8 bytes) 000547 1786839082.608305 read 0x016f86 thru 0x01b16e (0x41e9 bytes) 000548 1786839082.608900 mapread 0x006f57 thru 0x00f28a (0x8334 bytes) 000549 1786839082.609761 read 0x00e4bb thru 0x01733d (0x8e83 bytes) 000550 1786839082.610231 mapwrite 0x01216e thru 0x019f1b (0x7dae bytes) 000551 1786839082.630823 mapwrite 0x0153d8 thru 0x015dfd (0xa26 bytes) 000554 1786839082.647333 read 0x00f951 thru 0x018ae6 (0x9196 bytes) 000555 1786839082.647955 write 0x03ac00 thru 0x03ffff (0x5400 bytes) 000556 1786839082.649216 trunc from 0x040000 to 0x02dc1a (0x123e5 bytes) 000558 1786839082.655428 write 0x0245cb thru 0x02dd4f (0x9785 bytes) 000560 1786839082.661264 mapread 0x01459d thru 0x01e0a3 (0x9b07 bytes) 000561 1786839082.662518 mapwrite 0x03ffd8 thru 0x03ffff (0x28 bytes) 000562 1786839082.690824 trunc from 0x040000 to 0x00b7a0 (0x3485f bytes) 000564 1786839082.699097 read 0x005791 thru 0x0095c2 (0x3e32 bytes) 000567 1786839082.700539 read 0x0013ac thru 0x007bb1 (0x6806 bytes) 000568 1786839082.703038 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0xb7a1): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb7a1): Mirror 8: 0xf387450 (stale) Mirror 11: 0xf387450 (stale) Mirror 12: 0xf387450 (stale) Mirror 13: 0xf387450 (stale) Mirror 15: 0xf387450 (stale) Mirror 16: 0xf387450 (stale) Mirror 17: 0xf387450 (stale) Mirror 18: 0xf387450 (stale) Mirror 19: 0xf387450 (stale) Mirror 20: 0xf387450 (stale) Mirror 21: 0xbe6e0760 (stale) CRC-32 checksum value for chunk [0, 0xb7a1): Mirror 1: 0xbe6e0760 Mirror 22: 0xbe6e0760 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb7a1: skipped 000569 1786839083.048547 mapwrite 0x01bb52 thru 0x024078 (0x8527 bytes) 000570 1786839083.116791 write 0x02e469 thru 0x03132c (0x2ec4 bytes) 000574 1786839083.119544 mapwrite 0x032142 thru 0x03a42f (0x82ee bytes) 000578 1786839083.152501 trunc from 0x03a430 to 0x0236ae (0x16d81 bytes) 000579 1786839083.160224 write 0x003413 thru 0x00bb63 (0x8751 bytes) 000580 1786839083.166086 mapwrite 0x01ea12 thru 0x023ddd (0x53cc bytes) 000583 1786839083.196946 trunc from 0x023dde to 0x031bdf (0xde02 bytes) 000585 1786839083.203729 read 0x001c8a thru 0x008f02 (0x7279 bytes) 000586 1786839083.206141 mapwrite 0x030f66 thru 0x036cc0 (0x5d5b bytes) 000587 1786839083.234342 trunc from 0x036cc1 to 0x014214 (0x22aac bytes) 000588 1786839083.241445 write 0x007a92 thru 0x017933 (0xfea2 bytes) 000589 1786839083.243106 mapread 0x00af4e thru 0x015ab0 (0xab63 bytes) 000590 1786839083.244266 trunc from 0x017934 to 0x034f63 (0x1d630 bytes) 000592 1786839083.252526 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x34f64): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34f64): Mirror 8: 0x6a629807 (stale) Mirror 11: 0x6a629807 (stale) Mirror 12: 0x6a629807 (stale) Mirror 13: 0x6a629807 (stale) Mirror 15: 0x6a629807 (stale) Mirror 16: 0x6a629807 (stale) Mirror 17: 0x6a629807 (stale) Mirror 18: 0x6a629807 (stale) Mirror 19: 0x6a629807 (stale) Mirror 20: 0x6a629807 (stale) Mirror 21: 0xb1208ae7 (stale) Mirror 22: 0x6e0854bb (stale) CRC-32 checksum value for chunk [0, 0x34f64): Mirror 1: 0xfa80cb46 Mirror 23: 0xfa80cb46 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34f64: skipped 000594 1786839083.661427 write 0x0326e3 thru 0x034ca3 (0x25c1 bytes) 000597 1786839083.697730 mapread 0x031000 thru 0x034f63 (0x3f64 bytes) 000598 1786839083.701456 write 0x019506 thru 0x028d7c (0xf877 bytes) 000599 1786839083.707411 read 0x0059b7 thru 0x005dde (0x428 bytes) 000603 1786839083.712618 mapread 0x017859 thru 0x01e49f (0x6c47 bytes) 000604 1786839083.721373 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x34f64): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34f64): Mirror 8: 0x6a629807 (stale) Mirror 11: 0x6a629807 (stale) Mirror 12: 0x6a629807 (stale) Mirror 13: 0x6a629807 (stale) Mirror 15: 0x6a629807 (stale) Mirror 16: 0x6a629807 (stale) Mirror 17: 0x6a629807 (stale) Mirror 18: 0x6a629807 (stale) Mirror 19: 0x6a629807 (stale) Mirror 20: 0x6a629807 (stale) Mirror 21: 0xb1208ae7 (stale) Mirror 22: 0x6e0854bb (stale) Mirror 23: 0xfa80cb46 (stale) CRC-32 checksum value for chunk [0, 0x34f64): Mirror 1: 0x4cb60cd9 Mirror 24: 0x4cb60cd9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34f64: skipped 000606 1786839084.177629 mapread 0x000fca thru 0x00af37 (0x9f6e bytes) 000607 1786839084.191914 mapread 0x01dafc thru 0x02852c (0xaa31 bytes) 000610 1786839084.214887 mapread 0x016cac thru 0x01cf41 (0x6296 bytes) 000612 1786839084.227199 mapread 0x02235d thru 0x02e1ce (0xbe72 bytes) 000613 1786839084.237601 write 0x035ef2 thru 0x0360b1 (0x1c0 bytes) 000614 1786839084.263615 trunc from 0x0360b2 to 0x013ce7 (0x223ca bytes) 000616 1786839084.276657 read 0x007015 thru 0x009bee (0x2bda bytes) 000617 1786839084.280510 MIRROR_SPLIT mirror 11 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13ce8): Mirror 8: 0x84db879c (stale) Mirror 11: 0x84db879c (stale) Mirror 12: 0x84db879c (stale) Mirror 13: 0x84db879c (stale) Mirror 15: 0x84db879c (stale) Mirror 16: 0x84db879c (stale) Mirror 17: 0x84db879c (stale) Mirror 18: 0x84db879c (stale) Mirror 19: 0x84db879c (stale) Mirror 20: 0x84db879c (stale) Mirror 22: 0x9192c41a (stale) Mirror 23: 0xaa8f1281 (stale) Mirror 24: 0xaa8f1281 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13ce8): Mirror 8: 0x84db879c (stale) Mirror 11: 0x84db879c (stale) Mirror 12: 0x84db879c (stale) Mirror 13: 0x84db879c (stale) Mirror 15: 0x84db879c (stale) Mirror 16: 0x84db879c (stale) Mirror 17: 0x84db879c (stale) Mirror 18: 0x84db879c (stale) Mirror 19: 0x84db879c (stale) Mirror 20: 0x84db879c (stale) Mirror 22: 0x9192c41a (stale) Mirror 23: 0xaa8f1281 (stale) Mirror 24: 0xaa8f1281 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13ce8: skipped 000619 1786839084.534272 MIRROR_SPLIT mirror 2 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13ce8): Mirror 8: 0x84db879c (stale) Mirror 12: 0x84db879c (stale) Mirror 13: 0x84db879c (stale) Mirror 15: 0x84db879c (stale) Mirror 16: 0x84db879c (stale) Mirror 17: 0x84db879c (stale) Mirror 18: 0x84db879c (stale) Mirror 19: 0x84db879c (stale) Mirror 20: 0x84db879c (stale) Mirror 22: 0x9192c41a (stale) Mirror 23: 0xaa8f1281 (stale) Mirror 24: 0xaa8f1281 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13ce8): Mirror 8: 0x84db879c (stale) Mirror 12: 0x84db879c (stale) Mirror 13: 0x84db879c (stale) Mirror 15: 0x84db879c (stale) Mirror 16: 0x84db879c (stale) Mirror 17: 0x84db879c (stale) Mirror 18: 0x84db879c (stale) Mirror 19: 0x84db879c (stale) Mirror 20: 0x84db879c (stale) Mirror 22: 0x9192c41a (stale) Mirror 23: 0xaa8f1281 (stale) Mirror 24: 0xaa8f1281 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13ce8: skipped 000621 1786839084.809343 read 0x006fa6 thru 0x00903d (0x2098 bytes) 000622 1786839084.812474 trunc from 0x013ce8 to 0x031736 (0x1da4f bytes) 000623 1786839084.820112 read 0x01727b thru 0x01a01a (0x2da0 bytes) 000625 1786839084.825888 read 0x002e5d thru 0x003069 (0x20d bytes) 000626 1786839084.826467 mapwrite 0x010678 thru 0x0145af (0x3f38 bytes) 000628 1786839084.852532 trunc from 0x031737 to 0x014e9f (0x1c897 bytes) 000630 1786839084.868624 mapwrite 0x013e77 thru 0x0239d0 (0xfb5a bytes) 000631 1786839084.909596 write 0x032c58 thru 0x039cc2 (0x706b bytes) 000633 1786839084.912564 read 0x01367a thru 0x01a311 (0x6c98 bytes) 000635 1786839084.914097 mapwrite 0x03040d thru 0x030668 (0x25c bytes) 000636 1786839084.951375 trunc from 0x039cc3 to 0x0084d8 (0x317ea bytes) 000637 1786839084.963105 mapread 0x007e2f thru 0x0084d8 (0x6aa bytes) 000638 1786839084.964061 mapread 0x005ec4 thru 0x0084d8 (0x2615 bytes) 000639 1786839084.968349 MIRROR_SPLIT mirror 8 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x84d9): Mirror 8: 0xef6a5cd6 (stale) Mirror 12: 0xef6a5cd6 (stale) Mirror 13: 0xef6a5cd6 (stale) Mirror 15: 0xef6a5cd6 (stale) Mirror 16: 0xef6a5cd6 (stale) Mirror 17: 0xef6a5cd6 (stale) Mirror 18: 0xef6a5cd6 (stale) Mirror 20: 0xef6a5cd6 (stale) Mirror 22: 0xef6a5cd6 (stale) Mirror 23: 0x545f1b9b (stale) Mirror 24: 0x545f1b9b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x84d9): Mirror 8: 0xef6a5cd6 (stale) Mirror 12: 0xef6a5cd6 (stale) Mirror 13: 0xef6a5cd6 (stale) Mirror 15: 0xef6a5cd6 (stale) Mirror 16: 0xef6a5cd6 (stale) Mirror 17: 0xef6a5cd6 (stale) Mirror 18: 0xef6a5cd6 (stale) Mirror 20: 0xef6a5cd6 (stale) Mirror 22: 0xef6a5cd6 (stale) Mirror 23: 0x545f1b9b (stale) Mirror 24: 0x545f1b9b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x84d9: skipped 000640 1786839085.227071 trunc from 0x0084d9 to 0x02c8ab (0x243d3 bytes) 000641 1786839085.233034 read 0x00fb3d thru 0x013cb8 (0x417c bytes) 000643 1786839085.237251 write 0x016138 thru 0x02187a (0xb743 bytes) 000645 1786839085.243913 MIRROR_RESYNC 12 mirrors 000648 1786839086.359252 write 0x033999 thru 0x03ffff (0xc667 bytes) 000649 1786839086.384988 read 0x01a261 thru 0x026fdb (0xcd7b bytes) 000651 1786839086.388702 mapread 0x03f87c thru 0x03ffff (0x784 bytes) 000652 1786839086.389592 write 0x02f90d thru 0x031baf (0x22a3 bytes) 000656 1786839086.394591 mapread 0x0225d0 thru 0x02469e (0x20cf bytes) 000658 1786839086.395747 write 0x01e74b thru 0x027fba (0x9870 bytes) 000659 1786839086.398623 trunc from 0x040000 to 0x02b826 (0x147d9 bytes) 000661 1786839086.406859 mapwrite 0x013421 thru 0x01d231 (0x9e11 bytes) 000662 1786839086.433652 read 0x024a4c thru 0x02b826 (0x6ddb bytes) 000664 1786839086.438255 write 0x027112 thru 0x02bf2a (0x4e19 bytes) 000666 1786839086.439349 read 0x0084db thru 0x012c48 (0xa76e bytes) 000667 1786839086.439818 trunc from 0x02bf2b to 0x01177a (0x1a7b0 bytes) 000668 1786839086.445313 read 0x004261 thru 0x011354 (0xd0f4 bytes) 000670 1786839086.446530 read 0x00b4c8 thru 0x00efa6 (0x3adf bytes) 000672 1786839086.448898 MIRROR_SPLIT mirror 6 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1177b): Mirror 8: 0x3ac2e0c (stale) Mirror 12: 0x3ac2e0c (stale) Mirror 13: 0x3ac2e0c (stale) Mirror 15: 0x3ac2e0c (stale) Mirror 16: 0x3ac2e0c (stale) Mirror 18: 0x3ac2e0c (stale) Mirror 20: 0x3ac2e0c (stale) Mirror 22: 0x3ac2e0c (stale) Mirror 23: 0x3ac2e0c (stale) Mirror 24: 0x3ac2e0c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1177b): Mirror 8: 0x3ac2e0c (stale) Mirror 12: 0x3ac2e0c (stale) Mirror 13: 0x3ac2e0c (stale) Mirror 15: 0x3ac2e0c (stale) Mirror 16: 0x3ac2e0c (stale) Mirror 18: 0x3ac2e0c (stale) Mirror 20: 0x3ac2e0c (stale) Mirror 22: 0x3ac2e0c (stale) Mirror 23: 0x3ac2e0c (stale) Mirror 24: 0x3ac2e0c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1177b: skipped 000675 1786839086.643076 mapread 0x00b9c2 thru 0x01177a (0x5db9 bytes) 000676 1786839086.651893 trunc from 0x01177b to 0x02114d (0xf9d3 bytes) 000677 1786839086.658083 trunc from 0x02114e to 0x0128ba (0xe893 bytes) 000681 1786839086.666625 read 0x005fa2 thru 0x008ad4 (0x2b33 bytes) 000683 1786839086.669774 mapwrite 0x03976e thru 0x03ffff (0x6892 bytes) 000685 1786839086.697157 write 0x0295e9 thru 0x02e4d0 (0x4ee8 bytes) 000687 1786839086.702246 read 0x02a663 thru 0x02bc17 (0x15b5 bytes) 000688 1786839086.702725 read 0x01b756 thru 0x0270d3 (0xb97e bytes) 000690 1786839086.707020 write 0x015d9f thru 0x01fd7c (0x9fde bytes) 000691 1786839086.707925 mapwrite 0x0357bd thru 0x03ffff (0xa843 bytes) 000692 1786839086.725851 write 0x00c850 thru 0x0124c5 (0x5c76 bytes) 000693 1786839086.726785 read 0x0046b2 thru 0x01326b (0xebba bytes) 000696 1786839086.728130 read 0x018516 thru 0x02595c (0xd447 bytes) 000697 1786839086.728646 mapwrite 0x0185e4 thru 0x023d36 (0xb753 bytes) 000700 1786839086.746350 write 0x027e7a thru 0x0304ac (0x8633 bytes) 000701 1786839086.747379 mapwrite 0x02c8cb thru 0x0371e2 (0xa918 bytes) 000702 1786839086.771549 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x72adf40f (stale) Mirror 12: 0x72adf40f (stale) Mirror 13: 0x72adf40f (stale) Mirror 15: 0x72adf40f (stale) Mirror 16: 0x72adf40f (stale) Mirror 18: 0x72adf40f (stale) Mirror 22: 0x72adf40f (stale) Mirror 23: 0x72adf40f (stale) Mirror 24: 0x72adf40f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x72adf40f (stale) Mirror 12: 0x72adf40f (stale) Mirror 13: 0x72adf40f (stale) Mirror 15: 0x72adf40f (stale) Mirror 16: 0x72adf40f (stale) Mirror 18: 0x72adf40f (stale) Mirror 22: 0x72adf40f (stale) Mirror 23: 0x72adf40f (stale) Mirror 24: 0x72adf40f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000704 1786839086.963754 mapwrite 0x02465e thru 0x024b4d (0x4f0 bytes) 000705 1786839086.979358 read 0x038f75 thru 0x03ffff (0x708b bytes) 000707 1786839086.983126 read 0x01b9be thru 0x01fe2a (0x446d bytes) 000708 1786839086.987351 read 0x016b4c thru 0x0215eb (0xaaa0 bytes) 000710 1786839086.988400 mapwrite 0x024de4 thru 0x02cd47 (0x7f64 bytes) 000715 1786839087.008808 mapread 0x025a24 thru 0x03059f (0xab7c bytes) 000716 1786839087.009954 trunc from 0x040000 to 0x036ce0 (0x931f bytes) 000718 1786839087.015866 mapwrite 0x02243b thru 0x0319d9 (0xf59f bytes) 000719 1786839087.039946 mapwrite 0x00819b thru 0x00bc52 (0x3ab8 bytes) 000720 1786839087.056892 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36ce1): Mirror 8: 0xc039f56 (stale) Mirror 12: 0xc039f56 (stale) Mirror 15: 0xc039f56 (stale) Mirror 16: 0xc039f56 (stale) Mirror 18: 0xc039f56 (stale) Mirror 22: 0xc039f56 (stale) Mirror 23: 0xc039f56 (stale) Mirror 24: 0xc039f56 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ce1): Mirror 8: 0xc039f56 (stale) Mirror 12: 0xc039f56 (stale) Mirror 15: 0xc039f56 (stale) Mirror 16: 0xc039f56 (stale) Mirror 18: 0xc039f56 (stale) Mirror 22: 0xc039f56 (stale) Mirror 23: 0xc039f56 (stale) Mirror 24: 0xc039f56 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ce1: skipped 000721 1786839087.249381 trunc from 0x036ce1 to 0x016bd0 (0x20110 bytes) 000722 1786839087.255949 write 0x00101a thru 0x0045fc (0x35e3 bytes) 000724 1786839087.261397 mapwrite 0x02de16 thru 0x038a66 (0xac51 bytes) 000727 1786839087.303197 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x38a67): Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38a67): Mirror 8: 0xbb6840ef (stale) Mirror 12: 0xbb6840ef (stale) Mirror 15: 0xbb6840ef (stale) Mirror 16: 0xbb6840ef (stale) Mirror 18: 0xbb6840ef (stale) Mirror 22: 0xbb6840ef (stale) Mirror 23: 0xbb6840ef (stale) Mirror 24: 0xbb6840ef (stale) CRC-32 checksum value for chunk [0, 0x38a67): Mirror 1: 0x8b4dd7dd Mirror 25: 0x8b4dd7dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38a67: skipped 000729 1786839087.700651 write 0x037428 thru 0x03b88a (0x4463 bytes) 000730 1786839087.729156 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b88b): Mirror 8: 0x76486b00 (stale) Mirror 12: 0x76486b00 (stale) Mirror 15: 0x76486b00 (stale) Mirror 16: 0x76486b00 (stale) Mirror 18: 0x76486b00 (stale) Mirror 22: 0x76486b00 (stale) Mirror 23: 0x76486b00 (stale) Mirror 24: 0x76486b00 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b88b): Mirror 8: 0x76486b00 (stale) Mirror 12: 0x76486b00 (stale) Mirror 15: 0x76486b00 (stale) Mirror 16: 0x76486b00 (stale) Mirror 18: 0x76486b00 (stale) Mirror 22: 0x76486b00 (stale) Mirror 23: 0x76486b00 (stale) Mirror 24: 0x76486b00 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b88b: skipped 000731 1786839087.962209 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b88b): Mirror 12: 0x76486b00 (stale) Mirror 15: 0x76486b00 (stale) Mirror 16: 0x76486b00 (stale) Mirror 18: 0x76486b00 (stale) Mirror 22: 0x76486b00 (stale) Mirror 23: 0x76486b00 (stale) Mirror 24: 0x76486b00 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b88b): Mirror 12: 0x76486b00 (stale) Mirror 15: 0x76486b00 (stale) Mirror 16: 0x76486b00 (stale) Mirror 18: 0x76486b00 (stale) Mirror 22: 0x76486b00 (stale) Mirror 23: 0x76486b00 (stale) Mirror 24: 0x76486b00 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b88b: skipped 000734 1786839088.144735 mapread 0x0311af thru 0x03453d (0x338f bytes) 000739 1786839088.158272 MIRROR_RESYNC 8 mirrors 000740 1786839088.972436 mapwrite 0x02a63b thru 0x0368a7 (0xc26d bytes) 000742 1786839089.043521 mapwrite 0x025403 thru 0x027cdf (0x28dd bytes) 000749 1786839089.064285 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3b88b): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b88b): Mirror 12: 0x4f460da2 (stale) Mirror 15: 0x4f460da2 (stale) Mirror 16: 0x4f460da2 (stale) Mirror 18: 0x4f460da2 (stale) Mirror 22: 0x4f460da2 (stale) Mirror 23: 0x4f460da2 (stale) Mirror 24: 0x4f460da2 (stale) CRC-32 checksum value for chunk [0, 0x3b88b): Mirror 1: 0x2b890ee7 Mirror 25: 0x2b890ee7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b88b: skipped 000750 1786839089.503301 MIRROR_RESYNC 9 mirrors 000752 1786839090.750444 mapwrite 0x0117f1 thru 0x011d5c (0x56c bytes) 000756 1786839090.811451 read 0x01a841 thru 0x028b91 (0xe351 bytes) 000757 1786839090.817117 mapread 0x01e9f9 thru 0x023472 (0x4a7a bytes) 000758 1786839090.818218 mapread 0x019d44 thru 0x0240e5 (0xa3a2 bytes) 000761 1786839090.824436 trunc from 0x03b88b to 0x017103 (0x24787 bytes) 000762 1786839090.839159 MIRROR_RESYNC 9 mirrors 000763 1786839091.815720 mapread 0x011bcf thru 0x017103 (0x5535 bytes) 000764 1786839091.829032 trunc from 0x017104 to 0x026c56 (0xfb53 bytes) 000766 1786839091.873053 read 0x019f05 thru 0x01ca7f (0x2b7b bytes) 000767 1786839091.878554 mapread 0x016900 thru 0x022e1f (0xc520 bytes) 000768 1786839091.887698 write 0x010c2c thru 0x01e812 (0xdbe7 bytes) 000769 1786839091.888889 mapread 0x00e98c thru 0x01810e (0x9783 bytes) 000770 1786839091.893736 mapread 0x013581 thru 0x01c774 (0x91f4 bytes) 000773 1786839091.895952 write 0x0175e1 thru 0x019d93 (0x27b3 bytes) 000776 1786839091.898122 write 0x003e91 thru 0x0058a1 (0x1a11 bytes) 000778 1786839091.905643 read 0x0260ab thru 0x026c56 (0xbac bytes) 000779 1786839091.906300 read 0x003d8e thru 0x006947 (0x2bba bytes) 000782 1786839091.912265 write 0x024710 thru 0x025ea2 (0x1793 bytes) 000783 1786839091.913377 mapread 0x017628 thru 0x01b563 (0x3f3c bytes) 000784 1786839091.915191 trunc from 0x026c57 to 0x03371a (0xcac4 bytes) 000785 1786839091.925038 mapread 0x01a534 thru 0x021c4a (0x7717 bytes) 000787 1786839091.926544 write 0x0093fb thru 0x01355c (0xa162 bytes) 000792 1786839091.929625 mapwrite 0x019bbf thru 0x01c214 (0x2656 bytes) 000793 1786839091.955103 write 0x002e78 thru 0x006ae8 (0x3c71 bytes) 000795 1786839091.958902 MIRROR_RESYNC 9 mirrors 000796 1786839092.657824 mapwrite 0x02167a thru 0x022caa (0x1631 bytes) 000798 1786839092.713413 trunc from 0x03371b to 0x002916 (0x30e04 bytes) 000799 1786839092.724468 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2917): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2917): Mirror 12: 0x2dccb1fd (stale) Mirror 15: 0x2dccb1fd (stale) Mirror 16: 0x2dccb1fd (stale) Mirror 18: 0x2dccb1fd (stale) Mirror 22: 0x2dccb1fd (stale) Mirror 23: 0x2dccb1fd (stale) Mirror 24: 0x2dccb1fd (stale) Mirror 25: 0x2dccb1fd (stale) CRC-32 checksum value for chunk [0, 0x2917): Mirror 1: 0x2dccb1fd Mirror 26: 0x2dccb1fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2917: skipped 000800 1786839093.033332 mapwrite 0x02e97a thru 0x03687f (0x7f06 bytes) 000801 1786839093.099436 mapread 0x010a65 thru 0x015d00 (0x529c bytes) 000802 1786839093.105116 mapread 0x020998 thru 0x024bc8 (0x4231 bytes) 000803 1786839093.110905 mapwrite 0x015aed thru 0x016255 (0x769 bytes) 000806 1786839093.127737 mapwrite 0x0363e2 thru 0x03ffff (0x9c1e bytes) 000808 1786839093.159873 MIRROR_RESYNC 10 mirrors 000810 1786839094.235026 trunc from 0x040000 to 0x01f5a8 (0x20a57 bytes) 000811 1786839094.284148 read 0x014cb5 thru 0x01618f (0x14db bytes) 000812 1786839094.287695 trunc from 0x01f5a9 to 0x030b74 (0x115cc bytes) 000813 1786839094.299960 read 0x021dce thru 0x029b6d (0x7da0 bytes) 000814 1786839094.308507 mapwrite 0x00604a thru 0x00fedf (0x9e96 bytes) 000817 1786839094.340292 trunc from 0x030b75 to 0x03a67d (0x9b09 bytes) 000819 1786839094.355384 read 0x010f60 thru 0x015430 (0x44d1 bytes) 000820 1786839094.356123 trunc from 0x03a67e to 0x038e6f (0x180e bytes) 000821 1786839094.367125 read 0x028d55 thru 0x0368bf (0xdb6b bytes) 000822 1786839094.373136 trunc from 0x038e70 to 0x020871 (0x185fe bytes) 000824 1786839094.383966 MIRROR_RESYNC 10 mirrors 000825 1786839095.525383 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) 000826 1786839095.675322 mapwrite 0x017249 thru 0x022bae (0xb966 bytes) 000828 1786839095.729140 trunc from 0x022baf to 0x018259 (0xa955 bytes) 000830 1786839095.736981 trunc from 0x01825a to 0x028c94 (0x10a3b bytes) 000831 1786839095.742716 mapread 0x01c0e3 thru 0x020441 (0x435f bytes) 000832 1786839095.746662 write 0x037256 thru 0x03e57e (0x7329 bytes) 000833 1786839095.749498 MIRROR_RESYNC 9 mirrors 000834 1786839096.764689 mapread 0x00bfbc thru 0x01b739 (0xf77e bytes) 000835 1786839096.784319 read 0x001669 thru 0x007a53 (0x63eb bytes) 000836 1786839096.791032 write 0x0311c6 thru 0x03dada (0xc915 bytes) 000839 1786839096.825201 mapwrite 0x00c7c5 thru 0x01a251 (0xda8d bytes) 000840 1786839096.861287 mapread 0x015bcc thru 0x02249d (0xc8d2 bytes) 000841 1786839096.868049 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e57f): Mirror 12: 0xbcd8144e (stale) Mirror 15: 0xbcd8144e (stale) Mirror 16: 0xbcd8144e (stale) Mirror 22: 0xbcd8144e (stale) Mirror 23: 0xbcd8144e (stale) Mirror 24: 0xbcd8144e (stale) Mirror 25: 0xbcd8144e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e57f): Mirror 12: 0xbcd8144e (stale) Mirror 15: 0xbcd8144e (stale) Mirror 16: 0xbcd8144e (stale) Mirror 22: 0xbcd8144e (stale) Mirror 23: 0xbcd8144e (stale) Mirror 24: 0xbcd8144e (stale) Mirror 25: 0xbcd8144e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e57f: skipped 000843 1786839097.056290 MIRROR_RESYNC 8 mirrors 000844 1786839097.879662 write 0x010a7c thru 0x01e559 (0xdade bytes) 000845 1786839097.906331 trunc from 0x03e57f to 0x0075e1 (0x36f9d bytes) 000846 1786839097.916030 mapwrite 0x00b7e7 thru 0x00ed3a (0x3554 bytes) 000847 1786839097.954879 read 0x00c9b5 thru 0x00ed3a (0x2386 bytes) 000849 1786839097.956239 mapwrite 0x03d170 thru 0x03ffff (0x2e90 bytes) 000850 1786839097.992464 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xddd327e5 (stale) Mirror 15: 0xddd327e5 (stale) Mirror 16: 0xddd327e5 (stale) Mirror 22: 0xddd327e5 (stale) Mirror 23: 0xddd327e5 (stale) Mirror 24: 0xddd327e5 (stale) Mirror 25: 0xddd327e5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x77072be5 Mirror 26: 0x77072be5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000854 1786839098.477102 read 0x01a9c2 thru 0x026419 (0xba58 bytes) 000858 1786839098.483092 mapwrite 0x00f4bc thru 0x014b6f (0x56b4 bytes) 000859 1786839098.559544 write 0x029bf5 thru 0x02c834 (0x2c40 bytes) 000860 1786839098.565088 mapwrite 0x024ccb thru 0x02746a (0x27a0 bytes) 000864 1786839098.590712 write 0x034220 thru 0x035b7f (0x1960 bytes) 000865 1786839098.597106 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xddd327e5 (stale) Mirror 15: 0xddd327e5 (stale) Mirror 16: 0xddd327e5 (stale) Mirror 23: 0xddd327e5 (stale) Mirror 24: 0xddd327e5 (stale) Mirror 25: 0xddd327e5 (stale) Mirror 26: 0x77072be5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xddd327e5 (stale) Mirror 15: 0xddd327e5 (stale) Mirror 16: 0xddd327e5 (stale) Mirror 23: 0xddd327e5 (stale) Mirror 24: 0xddd327e5 (stale) Mirror 25: 0xddd327e5 (stale) Mirror 26: 0x77072be5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000866 1786839098.792929 trunc from 0x040000 to 0x014803 (0x2b7fc bytes) 000868 1786839098.802745 write 0x0264a0 thru 0x032735 (0xc296 bytes) 000869 1786839098.804145 read 0x01cd9a thru 0x0231be (0x6425 bytes) 000870 1786839098.811098 mapwrite 0x027c90 thru 0x028572 (0x8e3 bytes) 000875 1786839098.844738 MIRROR_RESYNC 8 mirrors 000876 1786839100.058405 mapwrite 0x0105ba thru 0x01fd9c (0xf7e3 bytes) 000877 1786839100.152394 MIRROR_RESYNC 8 mirrors 000878 1786839101.486825 write 0x01a7b6 thru 0x02a3ea (0xfc35 bytes) 000879 1786839101.522116 mapwrite 0x009a99 thru 0x0135d2 (0x9b3a bytes) 000880 1786839101.564464 mapwrite 0x02ccda thru 0x03a540 (0xd867 bytes) 000881 1786839101.604406 mapwrite 0x012245 thru 0x0163cf (0x418b bytes) 000883 1786839101.633713 read 0x02566d thru 0x029095 (0x3a29 bytes) 000886 1786839101.635168 read 0x0134c7 thru 0x0226f2 (0xf22c bytes) 000887 1786839101.635870 trunc from 0x03a541 to 0x02f800 (0xad40 bytes) 000888 1786839101.645240 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2f801): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f801): Mirror 12: 0xaee53bc7 (stale) Mirror 15: 0xaee53bc7 (stale) Mirror 16: 0xaee53bc7 (stale) Mirror 23: 0xaee53bc7 (stale) Mirror 24: 0xaee53bc7 (stale) Mirror 25: 0xaee53bc7 (stale) Mirror 26: 0xaee53bc7 (stale) CRC-32 checksum value for chunk [0, 0x2f801): Mirror 1: 0x723688f1 Mirror 27: 0x723688f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f801: skipped 000889 1786839102.135755 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f801): Mirror 12: 0xaee53bc7 (stale) Mirror 15: 0xaee53bc7 (stale) Mirror 16: 0xaee53bc7 (stale) Mirror 24: 0xaee53bc7 (stale) Mirror 25: 0xaee53bc7 (stale) Mirror 26: 0xaee53bc7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f801): Mirror 12: 0xaee53bc7 (stale) Mirror 15: 0xaee53bc7 (stale) Mirror 16: 0xaee53bc7 (stale) Mirror 24: 0xaee53bc7 (stale) Mirror 25: 0xaee53bc7 (stale) Mirror 26: 0xaee53bc7 (stale) CRC-32 checksum value for chunk [0, 0x2f801): Mirror 1: 0x723688f1 Mirror 27: 0x723688f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f801: skipped 000890 1786839102.471178 trunc from 0x02f801 to 0x028bc2 (0x6c3e bytes) 000891 1786839102.506806 mapread 0x00551a thru 0x0090b0 (0x3b97 bytes) 000892 1786839102.520675 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28bc3): Mirror 15: 0x21c0fb4d (stale) Mirror 16: 0x21c0fb4d (stale) Mirror 24: 0x21c0fb4d (stale) Mirror 25: 0x21c0fb4d (stale) Mirror 26: 0x21c0fb4d (stale) Mirror 27: 0xc6578c43 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28bc3): Mirror 15: 0x21c0fb4d (stale) Mirror 16: 0x21c0fb4d (stale) Mirror 24: 0x21c0fb4d (stale) Mirror 25: 0x21c0fb4d (stale) Mirror 26: 0x21c0fb4d (stale) Mirror 27: 0xc6578c43 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28bc3: skipped 000893 1786839102.731203 trunc from 0x028bc3 to 0x0071cd (0x219f5 bytes) 000895 1786839102.741364 MIRROR_RESYNC 7 mirrors 000899 1786839103.343315 trunc from 0x0071ce to 0x02447a (0x1d2ad bytes) 000900 1786839103.386546 trunc from 0x02447b to 0x01a7cc (0x9cae bytes) 000901 1786839103.397239 mapread 0x00984a thru 0x00fc10 (0x63c7 bytes) 000905 1786839103.417197 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a7cd): Mirror 15: 0x51a14f04 (stale) Mirror 24: 0x51a14f04 (stale) Mirror 25: 0x51a14f04 (stale) Mirror 26: 0x51a14f04 (stale) Mirror 27: 0x51a14f04 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a7cd): Mirror 15: 0x51a14f04 (stale) Mirror 24: 0x51a14f04 (stale) Mirror 25: 0x51a14f04 (stale) Mirror 26: 0x51a14f04 (stale) Mirror 27: 0x51a14f04 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a7cd: skipped 000906 1786839103.618436 mapread 0x0094be thru 0x00cd80 (0x38c3 bytes) 000907 1786839103.627711 write 0x027ec4 thru 0x03204c (0xa189 bytes) 000911 1786839103.631070 trunc from 0x03204d to 0x036bc8 (0x4b7c bytes) 000914 1786839103.642363 mapwrite 0x02b3d8 thru 0x0344a4 (0x90cd bytes) 000915 1786839103.672239 mapread 0x020596 thru 0x026a4b (0x64b6 bytes) 000916 1786839103.679179 write 0x00fe9f thru 0x015a7e (0x5be0 bytes) 000917 1786839103.683055 mapread 0x02dbc2 thru 0x036bc8 (0x9007 bytes) 000919 1786839103.684789 read 0x00c823 thru 0x0118b1 (0x508f bytes) 000920 1786839103.685437 write 0x01718d thru 0x021809 (0xa67d bytes) 000922 1786839103.691010 trunc from 0x036bc9 to 0x0222a4 (0x14924 bytes) 000926 1786839103.709256 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x222a5): Mirror 15: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x222a5): Mirror 15: 0x3c1c40c6 (stale) Mirror 24: 0x3c1c40c6 (stale) Mirror 25: 0x3c1c40c6 (stale) Mirror 26: 0x3c1c40c6 (stale) Mirror 27: 0x3c1c40c6 (stale) CRC-32 checksum value for chunk [0, 0x222a5): Mirror 1: 0x7991714f Mirror 28: 0x7991714f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x222a5: skipped 000928 1786839104.412101 read 0x0063b6 thru 0x00c654 (0x629f bytes) 000930 1786839104.429931 mapwrite 0x014c9e thru 0x01c790 (0x7af3 bytes) 000931 1786839104.536646 read 0x018431 thru 0x020059 (0x7c29 bytes) 000932 1786839104.542336 mapread 0x0218ef thru 0x0222a4 (0x9b6 bytes) 000933 1786839104.543164 trunc from 0x0222a5 to 0x0142dc (0xdfc8 bytes) 000934 1786839104.564451 trunc from 0x0142dd to 0x01e0b4 (0x9dd8 bytes) 000935 1786839104.576270 trunc from 0x01e0b5 to 0x00f735 (0xe97f bytes) 000938 1786839104.588610 read 0x00ea1e thru 0x00ee23 (0x406 bytes) 000939 1786839104.591781 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xf736): Mirror 15: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf736): Mirror 15: 0x46f9b37c (stale) Mirror 24: 0x46f9b37c (stale) Mirror 25: 0x46f9b37c (stale) Mirror 26: 0x46f9b37c (stale) Mirror 27: 0x46f9b37c (stale) Mirror 28: 0x46f9b37c (stale) CRC-32 checksum value for chunk [0, 0xf736): Mirror 1: 0x46f9b37c Mirror 29: 0x46f9b37c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf736: skipped 000940 1786839105.184768 mapread 0x0048d5 thru 0x00c412 (0x7b3e bytes) 000941 1786839105.198885 mapread 0x0073fb thru 0x00f735 (0x833b bytes) 000942 1786839105.207921 mapread 0x000a88 thru 0x0040ca (0x3643 bytes) 000945 1786839105.242917 trunc from 0x00f736 to 0x03ebd5 (0x2f4a0 bytes) 000946 1786839105.296823 mapwrite 0x02a0bd thru 0x033738 (0x967c bytes) 000947 1786839105.350173 trunc from 0x03ebd6 to 0x012b39 (0x2c09c bytes) 000949 1786839105.366598 mapwrite 0x03b0d5 thru 0x03b831 (0x75d bytes) 000951 1786839105.417814 trunc from 0x03b832 to 0x0098da (0x31f57 bytes) 000954 1786839105.434004 write 0x007a5d thru 0x00b2b6 (0x385a bytes) 000957 1786839105.439836 mapwrite 0x018680 thru 0x022694 (0xa015 bytes) 000958 1786839105.515271 trunc from 0x022695 to 0x036b43 (0x144af bytes) 000959 1786839105.528500 MIRROR_RESYNC 8 mirrors 000960 1786839106.842961 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) 000963 1786839107.035032 write 0x01e51d thru 0x02de56 (0xf93a bytes) 000964 1786839107.074318 mapread 0x011d48 thru 0x019873 (0x7b2c bytes) 000965 1786839107.090726 mapwrite 0x007266 thru 0x011d7f (0xab1a bytes) 000966 1786839107.126309 mapread 0x034362 thru 0x036b43 (0x27e2 bytes) 000967 1786839107.132731 write 0x009784 thru 0x018926 (0xf1a3 bytes) 000968 1786839107.134190 mapread 0x022d2d thru 0x02dd83 (0xb057 bytes) 000969 1786839107.136929 read 0x027b11 thru 0x03672d (0xec1d bytes) 000970 1786839107.137562 trunc from 0x036b44 to 0x03a89f (0x3d5c bytes) 000972 1786839107.152780 trunc from 0x03a8a0 to 0x00d4dd (0x2d3c2 bytes) 000975 1786839107.165181 trunc from 0x00d4de to 0x0052a6 (0x8237 bytes) 000976 1786839107.174118 MIRROR_RESYNC 7 mirrors 000977 1786839107.830105 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) 000979 1786839108.005225 write 0x0258df thru 0x02954c (0x3c6e bytes) 000980 1786839108.037236 write 0x00d43f thru 0x015209 (0x7dcb bytes) 000981 1786839108.042274 mapwrite 0x03c6d9 thru 0x03ffff (0x3927 bytes) 000983 1786839108.084301 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 24: 0x65f63ced (stale) Mirror 25: 0x65f63ced (stale) Mirror 27: 0x65f63ced (stale) Mirror 28: 0x65f63ced (stale) Mirror 29: 0x65f63ced (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8a0759fd Mirror 30: 0x8a0759fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000984 1786839108.583567 mapread 0x0043f1 thru 0x00b06f (0x6c7f bytes) 000985 1786839108.599390 trunc from 0x040000 to 0x020dc8 (0x1f237 bytes) 000986 1786839108.643202 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x20dc9): Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20dc9): Mirror 24: 0x4b24cd27 (stale) Mirror 25: 0x4b24cd27 (stale) Mirror 27: 0x4b24cd27 (stale) Mirror 28: 0x4b24cd27 (stale) Mirror 29: 0x4b24cd27 (stale) Mirror 30: 0x6aa78adc (stale) CRC-32 checksum value for chunk [0, 0x20dc9): Mirror 1: 0x6aa78adc Mirror 31: 0x6aa78adc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20dc9: skipped 000988 1786839109.163887 mapwrite 0x0143f9 thru 0x023eb6 (0xfabe bytes) 000989 1786839109.253259 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x48d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23eb7): Mirror 24: 0xf142248e (stale) Mirror 25: 0xf142248e (stale) Mirror 27: 0xf142248e (stale) Mirror 28: 0xf142248e (stale) Mirror 29: 0xf142248e (stale) Mirror 31: 0x92f7a544 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23eb7): Mirror 24: 0xf142248e (stale) Mirror 25: 0xf142248e (stale) Mirror 27: 0xf142248e (stale) Mirror 28: 0xf142248e (stale) Mirror 29: 0xf142248e (stale) Mirror 31: 0x92f7a544 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23eb7: skipped 000992 1786839109.475706 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x23eb7): Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23eb7): Mirror 24: 0xf142248e (stale) Mirror 25: 0xf142248e (stale) Mirror 27: 0xf142248e (stale) Mirror 28: 0xf142248e (stale) Mirror 29: 0xf142248e (stale) Mirror 31: 0x92f7a544 (stale) CRC-32 checksum value for chunk [0, 0x23eb7): Mirror 1: 0x46b5ae37 Mirror 32: 0x46b5ae37 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23eb7: skipped 000995 1786839109.954461 trunc from 0x023eb7 to 0x03a1cf (0x16319 bytes) 001000 1786839110.011185 MIRROR_RESYNC 8 mirrors All operations completed A-OK! PASS 70a (59s) == sanity-flr test 71: check mirror extend parent fid ==== 20:11:53 (1786839113) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 72: check append on FLR file ========== 20:11:54 (1786839114) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6f4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0982409 s, 21.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6f9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6f4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07893 s, 13.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6f9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6f5:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6f4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6f9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6f5:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6f4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6fa:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (3s) == sanity-flr test 200a: stress test ===================== 20:11:57 (1786839117) Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre2 Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8383104 .. Extending file size to 2787808 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7989024 .. Extending file size to 8754528 .. Extending file size to 8110720 .. Extending file size to 9232864 .. Extending file size to 9256800 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 59359 59360 59361 59363 59366 failed 192.168.203.106@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg306-client.virtnet /mnt/lustre2 (opts:) 192.168.203.106@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg306-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (68s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:13:05 (1786839185) Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre2 Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5028352 .. Extending file size to 2122784 .. failed Extending file size to 6068192 .. Extending file size to 2246976 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6608160 .. failed Extending file size to 8250176 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 5358112 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6008928 .. failed Extending file size to 8619776 .. Extending file size to 8642560 .. Extending file size to 7593952 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8767520 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9053696 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9048576 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9096512 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 62117 62118 62120 62121 62123 192.168.203.106@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg306-client.virtnet /mnt/lustre2 (opts:) 192.168.203.106@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg306-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (66s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:14:11 (1786839251) Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3328 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3328 2204928 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0499412 s, 42.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.111157 s, 18.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x6439f40f (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.106@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg306-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:14:18 (1786839258) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0375589 s, 55.8 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 20:14:20 (1786839260) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4df:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x70b:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x70b:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x70c:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x70f:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:14:23 (1786839263) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0491741 s, 42.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x70d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x710:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x711:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:14:25 (1786839265) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0534099 s, 39.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x70e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x70f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x712:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x70e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x70f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x712:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x713:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:14:28 (1786839268) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0465742 s, 45.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x710:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x714:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x710:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x714:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x715:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:14:52 (1786839292) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg306-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286652 s, 36.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x711:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x717:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x716:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x711:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x717:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x716:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x712:0x0] } PASS 204d (44s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:15:36 (1786839336) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg306-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x713:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x714:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0447006 s, 46.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x713:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x714:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x718:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x713:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x714:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x718:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:16:22 (1786839382) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg306-server mds-ost sync done. Creating new pool test_204f oleg306-server: Pool lustre.test_204f created Adding targets to pool oleg306-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg306-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0764198 s, 27.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x715:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x719:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x715:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x719:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg306-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg306-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg306-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (59s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:17:21 (1786839441) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:17:23 (1786839443) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:17:26 (1786839446) oleg306-server: Pool lustre.test_206 created Adding targets to pool oleg306-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg306-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg306-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg306-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg306-server: Pool lustre.test_206 destroyed PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:17:41 (1786839461) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028134 s, 37.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135498 s, 77.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xa23a06ec Mirror 3: 0xa23a06ec PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:17:44 (1786839464) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:17:45 (1786839465) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:17:47 (1786839467) Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0356552 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1d40000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1410000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1410000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1410000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x25f0000 Read this (4 bytes): BBBB 192.168.203.106@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg306-client.virtnet /mnt/lustre2 (opts:) PASS 209a (16s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:18:03 (1786839483) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00330512 s, 2.5 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:18:06 (1786839486) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175794 s, 59.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x504 lmm_fid: [0x200000404:0x504:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x504 lmm_fid: [0x200000404:0x504:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x724:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835204 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-15 20:18:06.000000000 -0400 Modify: 2026-08-15 20:18:06.000000000 -0400 Change: 2026-08-15 20:18:07.000000000 -0400 Birth: 2026-08-15 20:18:06.000000000 -0400 PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:18:09 (1786839489) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:18:10 (1786839490) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.151901 s, 270 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0158387 s, 259 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 15 20:18 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x506:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 15 20:18 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:18:12 (1786839492) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (3s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:18:15 (1786839495) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (3s) == sanity-flr test complete, duration 944 sec ============ 20:18:18 (1786839498) === sanity-flr: start cleanup 20:18:19 (1786839499) === === sanity-flr: finish cleanup 20:18:20 (1786839500) ===