-----============= acceptance-small: sanity-flr ============----- Tue Aug 25 21:19:55 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg336-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 21:20:15 (1787707215) === oleg336-client.virtnet: executing check_config_client /mnt/lustre oleg336-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg336-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8976d184e000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8976d184e000.idle_timeout=debug disable quota as required oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 21:20:57 (1787707257) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7626] == sanity-flr test 0a: lfs mirror create with -N option == 21:21:00 (1787707260) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 21:21:18 (1787707278) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 21:21:21 (1787707281) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 21:21:24 (1787707284) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 21:21:49 (1787707309) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 21:21:52 (1787707312) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 21:21:56 (1787707316) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8976d184e000.stats=clear osc.lustre-OST0001-osc-ffff8976d184e000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 21:22:04 (1787707324) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 21:22:16 (1787707336) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 21:22:24 (1787707344) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 21:22:28 (1787707348) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 21:22:36 (1787707356) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 21:22:45 (1787707365) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 21:22:53 (1787707373) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 21:22:57 (1787707377) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 21:23:04 (1787707384) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 21:23:14 (1787707394) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 21:23:22 (1787707402) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 2.68929 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122715 s, 8.5 MB/s PASS 21 (14s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 21:23:36 (1787707416) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.376669 s, 27.8 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 21:23:46 (1787707426) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 21:23:57 (1787707437) Stopping /mnt/lustre-ost1 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8976d184e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8976d184e000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg336-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8976d184e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8976d184e000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg336-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (83s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 21:25:20 (1787707520) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.410398 s, 15.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8976d184e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8976d184e000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg336-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 21:26:04 (1787707564) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8976d184e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8976d184e000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg336-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8976d184e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8976d184e000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg336-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (81s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 21:27:25 (1787707645) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 7.09379 s, 7.4 MB/s -rw-r--r-- 1 root root 52428800 Aug 25 21:27 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.178s user 0m0.002s sys 0m0.168s Stopping /mnt/lustre-ost1 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8976d184e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8976d184e000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.212s user 0m0.000s sys 0m0.193s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg336-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8976d184e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8976d184e000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.188s user 0m0.002s sys 0m0.194s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg336-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg336-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8976d184e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8976d184e000.ost_server_uuid in FULL state after 0 sec PASS 33b (98s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 21:29:03 (1787707743) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 21:29:07 (1787707747) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 21:29:10 (1787707750) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 21:29:13 (1787707753) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 21:29:21 (1787707761) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.600753 s, 20.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 21:29:33 (1787707773) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526696 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233326 s, 44.9 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 21:29:41 (1787707781) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 21:29:49 (1787707789) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.001758 s, 582 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00138401 s, 740 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0011443 s, 895 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00147985 s, 692 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 21:30:01 (1787707801) create test files with size 15754000 .. d9f3e9bbd2579771849a2380bf2f4ef2 - e1bfaff356c7114e79ca169b481241a0 - 9f42cfcf698485e8793f1bd3fe14db6f - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg336-client.virtnet /mnt/lustre (opts:) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre PASS 37 (28s) == sanity-flr test 38: resync ============================ 21:30:29 (1787707829) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (19s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 21:30:48 (1787707848) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 21:30:56 (1787707856) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151676 s, 6.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575595 s, 18.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 21:31:06 (1787707866) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.105787 s, 19.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.319255 s, 13.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.181599 s, 11.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 21:31:23 (1787707883) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 21:31:27 (1787707887) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 21:31:30 (1787707890) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203147 s, 51.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 21:31:39 (1787707899) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 21:31:42 (1787707902) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 21:31:52 (1787707912) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.300453 s, 34.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg336-client.virtnet /mnt/lustre (opts:) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 21:32:02 (1787707922) llite.lustre-ffff8976c33cf800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.369604 s, 28.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg336-client.virtnet /mnt/lustre (opts:) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8976c3ad4800.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 21:32:17 (1787707937) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 21:32:55 (1787707975) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 21:33:29 (1787708009) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (5s) == sanity-flr test 48: Verify snapshot mirror ============ 21:33:34 (1787708014) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.34552 s, 9.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.229914 s, 13.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x38735dda (nosync,stale) Mirror 2: 0x38735dda (nosync,stale) ** verify mirror 2 doesn't change original checksum: 0a7de311a0fff7a7916aca5bae2f5691 - mirror 1 checksum: cb8209dbb1953bcbd14d9d532a14be27 - mirror 2 checksum: 0a7de311a0fff7a7916aca5bae2f5691 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-25T21:33:38-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-25T21:33:38-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 0a7de311a0fff7a7916aca5bae2f5691 - mirror 1 checksum: cb8209dbb1953bcbd14d9d532a14be27 - mirror 2 checksum: cb8209dbb1953bcbd14d9d532a14be27 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-25T21:33:38-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-25T21:33:38-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 48 (13s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 21:33:49 (1787708029) Failing mds1 on oleg336-server Stopping /mnt/lustre-mds1 (opts:) on oleg336-server 21:33:56 (1787708036) shut down facet: mds1 facet_host: oleg336-server facet_failover_host: oleg336-server Failover mds1 to oleg336-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-MDT0000 21:34:19 (1787708059) targets are mounted 21:34:19 (1787708059) facet_failover done oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559866 s, 18.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } PASS 50A (49s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 21:34:38 (1787708078) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (5s) == sanity-flr test 50b: mirror rsync handles sparseness == 21:34:43 (1787708083) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 21:34:47 (1787708087) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00620219 s, 2.6 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 21:34:56 (1787708096) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 21:35:00 (1787708100) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0079609 s, 515 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0745128 s, 55.0 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 21:35:08 (1787708108) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 21:35:33 (1787708133) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 21:36:03 (1787708163) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 21:36:27 (1787708187) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 21:36:54 (1787708214) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 21:37:03 (1787708223) mirror create pid 54331 mirror split pid 54333 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54331 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54333 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 21:38:10 (1787708290) Seed set to 8292 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1787708292.839782 mapwrite 0x03c4ee thru 0x03ffff (0x3b12 bytes) truncating to largest ever: 0x6cbf 000006 1787708292.957515 trunc from 0x040000 to 0x006cbe (0x39341 bytes) 000007 1787708292.975772 mapwrite 0x02785d thru 0x032c79 (0xb41d bytes) 000008 1787708293.059942 read 0x02c260 thru 0x032c79 (0x6a1a bytes) truncating to largest ever: 0x15398 000009 1787708293.060625 trunc from 0x032c7a to 0x015397 (0x1d8e2 bytes) truncating to largest ever: 0x27027 000013 1787708293.086707 trunc from 0x015398 to 0x027026 (0x11c8f bytes) 000014 1787708293.107111 mapwrite 0x01e9b8 thru 0x022e17 (0x4460 bytes) 000015 1787708293.182626 MIRROR_EXTEND to 2 mirrors 000019 1787708294.576034 read 0x01f512 thru 0x027026 (0x7b15 bytes) 000020 1787708294.597930 mapwrite 0x02e193 thru 0x037452 (0x92c0 bytes) 000021 1787708294.947846 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) 000022 1787708295.143181 mapwrite 0x01455f thru 0x0202bf (0xbd61 bytes) 000023 1787708295.229721 write 0x015826 thru 0x0228b3 (0xd08e bytes) 000024 1787708295.231302 mapread 0x000cbf thru 0x006d51 (0x6093 bytes) 000025 1787708295.265434 mapread 0x0050cc thru 0x00e827 (0x975c bytes) 000026 1787708295.269398 read 0x019d1c thru 0x01fcbc (0x5fa1 bytes) 000027 1787708295.271015 mapwrite 0x0287e7 thru 0x02e384 (0x5b9e bytes) 000028 1787708295.357946 MIRROR_EXTEND to 2 mirrors 000029 1787708296.549896 mapwrite 0x015739 thru 0x017584 (0x1e4c bytes) 000030 1787708296.822319 mapread 0x01f57c thru 0x0238cb (0x4350 bytes) truncating to largest ever: 0x2b857 000031 1787708296.833593 trunc from 0x037453 to 0x02b856 (0xbbfc bytes) 000032 1787708296.887572 MIRROR_RESYNC 2 mirrors 000033 1787708297.993387 MIRROR_RESYNC 2 mirrors 000035 1787708298.366029 mapwrite 0x002a5f thru 0x01220b (0xf7ad bytes) 000036 1787708298.678582 mapread 0x00f789 thru 0x017d09 (0x8581 bytes) 000038 1787708298.683879 mapwrite 0x037a2a thru 0x03ffff (0x85d6 bytes) 000040 1787708298.785728 mapwrite 0x03c821 thru 0x03ffff (0x37df bytes) 000041 1787708298.828812 mapread 0x03164e thru 0x03ffff (0xe9b2 bytes) 000042 1787708298.830113 trunc from 0x040000 to 0x01f850 (0x207af bytes) 000048 1787708298.851399 read 0x01ce50 thru 0x01d3fe (0x5af bytes) 000049 1787708298.856834 read 0x01b28c thru 0x01f850 (0x45c5 bytes) 000050 1787708298.865122 mapwrite 0x015d56 thru 0x021c9e (0xbf49 bytes) 000051 1787708298.969897 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) 000053 1787708299.183804 mapwrite 0x0084e9 thru 0x017089 (0xeba1 bytes) 000054 1787708299.293491 write 0x01419f thru 0x020854 (0xc6b6 bytes) 000055 1787708299.300059 read 0x00e01e thru 0x0189d3 (0xa9b6 bytes) 000056 1787708299.304516 read 0x01381a thru 0x0173e2 (0x3bc9 bytes) 000057 1787708299.315049 MIRROR_EXTEND to 2 mirrors 000061 1787708300.161084 trunc from 0x021c9f to 0x00c39f (0x158ff bytes) 000064 1787708300.254766 mapread 0x007d80 thru 0x009fb5 (0x2236 bytes) 000067 1787708300.274363 write 0x01f495 thru 0x02da42 (0xe5ae bytes) 000068 1787708300.276696 trunc from 0x02da43 to 0x01fb40 (0xdf02 bytes) 000069 1787708300.308280 mapread 0x014bca thru 0x01f3f7 (0xa82e bytes) 000070 1787708300.325926 MIRROR_RESYNC 2 mirrors 000072 1787708301.585426 mapread 0x01b6bb thru 0x01d5ac (0x1ef2 bytes) 000073 1787708301.634419 mapwrite 0x0351f1 thru 0x03ffff (0xae0f bytes) 000074 1787708301.896048 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa507908a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdea37880 Mirror 3: 0xdea37880 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000076 1787708303.474600 trunc from 0x040000 to 0x011d66 (0x2e299 bytes) 000077 1787708303.577070 read 0x00c941 thru 0x011d66 (0x5426 bytes) 000081 1787708303.592975 mapwrite 0x037341 thru 0x037d50 (0xa10 bytes) 000082 1787708303.722711 trunc from 0x037d51 to 0x0171bd (0x20b93 bytes) 000084 1787708303.771961 read 0x007765 thru 0x00d652 (0x5eee bytes) 000086 1787708303.787940 mapread 0x00bd8f thru 0x0171bd (0xb42f bytes) 000087 1787708303.789743 read 0x004e56 thru 0x00a5b6 (0x5761 bytes) 000089 1787708303.792986 write 0x0221f5 thru 0x0317b3 (0xf5bf bytes) 000091 1787708303.805841 read 0x02bba5 thru 0x0317b3 (0x5c0f bytes) 000095 1787708303.821307 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x317b4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x317b4): Mirror 2: 0x135327d8 (stale) Mirror 3: 0x135327d8 (stale) CRC-32 checksum value for chunk [0, 0x317b4): Mirror 1: 0xd94166eb Mirror 4: 0xd94166eb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x317b4: skipped 000098 1787708305.258111 mapread 0x0178d7 thru 0x01d027 (0x5751 bytes) 000100 1787708305.280895 trunc from 0x0317b4 to 0x011ed4 (0x1f8df bytes) 000102 1787708305.397850 write 0x026c59 thru 0x028180 (0x1528 bytes) 000103 1787708305.400309 write 0x0225fe thru 0x0303d0 (0xddd3 bytes) 000104 1787708305.420136 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x303d1): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x303d1): Mirror 2: 0xd74436e7 (stale) Mirror 3: 0xd74436e7 (stale) Mirror 4: 0x63df870f (stale) CRC-32 checksum value for chunk [0, 0x303d1): Mirror 1: 0x88b37a31 Mirror 5: 0x88b37a31 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x303d1: skipped 000106 1787708307.267098 mapwrite 0x016863 thru 0x01a142 (0x38e0 bytes) 000107 1787708307.440041 mapread 0x02ae96 thru 0x0303d0 (0x553b bytes) 000109 1787708307.465930 mapwrite 0x030f9a thru 0x033b8b (0x2bf2 bytes) 000111 1787708307.564931 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33b8c): Mirror 3: 0x729a6e29 (stale) Mirror 4: 0x6955da8c (stale) Mirror 5: 0x90f32d3e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33b8c): Mirror 3: 0x729a6e29 (stale) Mirror 4: 0x6955da8c (stale) Mirror 5: 0x90f32d3e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33b8c: skipped 000112 1787708308.011210 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x33b8c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33b8c): Mirror 3: 0x729a6e29 (stale) Mirror 4: 0x6955da8c (stale) Mirror 5: 0x90f32d3e (stale) CRC-32 checksum value for chunk [0, 0x33b8c): Mirror 1: 0xdc98448 Mirror 6: 0xdc98448 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33b8c: skipped 000114 1787708309.326273 MIRROR_RESYNC 5 mirrors 000116 1787708311.414815 mapwrite 0x012d4c thru 0x017b5c (0x4e11 bytes) 000117 1787708311.689644 read 0x00fdeb thru 0x01f7e4 (0xf9fa bytes) 000119 1787708311.701662 read 0x005218 thru 0x009e14 (0x4bfd bytes) 000120 1787708311.712928 mapread 0x0062c4 thru 0x011787 (0xb4c4 bytes) 000123 1787708311.716417 mapread 0x011b2a thru 0x01c623 (0xaafa bytes) 000125 1787708311.719761 mapwrite 0x03b68b thru 0x03ffff (0x4975 bytes) 000126 1787708311.788113 write 0x01e449 thru 0x0223be (0x3f76 bytes) 000127 1787708311.790031 mapwrite 0x010292 thru 0x011902 (0x1671 bytes) 000128 1787708311.835861 mapread 0x019d2a thru 0x01d70f (0x39e6 bytes) 000130 1787708311.837193 read 0x026286 thru 0x02ed09 (0x8a84 bytes) 000131 1787708311.837804 write 0x031f61 thru 0x03be9c (0x9f3c bytes) 000133 1787708311.840722 mapwrite 0x0395b6 thru 0x039f91 (0x9dc bytes) 000134 1787708311.883037 write 0x032f1d thru 0x037362 (0x4446 bytes) 000135 1787708311.885495 read 0x00a0cb thru 0x01844f (0xe385 bytes) 000136 1787708311.886792 mapwrite 0x013627 thru 0x01b0ea (0x7ac4 bytes) 000138 1787708311.935513 mapwrite 0x018efe thru 0x01e7c0 (0x58c3 bytes) 000139 1787708311.978532 read 0x01f0ba thru 0x02667a (0x75c1 bytes) 000142 1787708311.980242 mapread 0x0035f2 thru 0x012ca7 (0xf6b6 bytes) 000144 1787708311.983992 trunc from 0x040000 to 0x025c7a (0x1a385 bytes) 000146 1787708312.001500 mapread 0x009d3b thru 0x015913 (0xbbd9 bytes) 000147 1787708312.002992 write 0x036391 thru 0x03ff71 (0x9be1 bytes) truncating to largest ever: 0x38351 000148 1787708312.006795 trunc from 0x03ff72 to 0x038350 (0x7c21 bytes) 000150 1787708312.029245 mapread 0x02aaf7 thru 0x02f483 (0x498d bytes) 000153 1787708312.071495 mapread 0x02aa44 thru 0x02dee2 (0x349f bytes) 000154 1787708312.073956 write 0x0074b0 thru 0x010bfd (0x974e bytes) 000155 1787708312.097480 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x38351): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38351): Mirror 3: 0x17955b79 (stale) Mirror 4: 0x17955b79 (stale) Mirror 5: 0x17955b79 (stale) Mirror 6: 0x17955b79 (stale) CRC-32 checksum value for chunk [0, 0x38351): Mirror 1: 0xfb107cd2 Mirror 7: 0xfb107cd2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38351: skipped 000157 1787708313.408360 read 0x000cac thru 0x00cbbb (0xbf10 bytes) 000158 1787708313.434864 MIRROR_RESYNC 6 mirrors 000159 1787708315.532814 MIRROR_RESYNC 6 mirrors 000160 1787708316.075665 trunc from 0x038351 to 0x00c978 (0x2b9d8 bytes) 000161 1787708316.164859 mapwrite 0x038a83 thru 0x03ffff (0x757d bytes) 000162 1787708316.281185 read 0x0338df thru 0x034372 (0xa94 bytes) 000164 1787708316.282442 read 0x011870 thru 0x020818 (0xefa9 bytes) 000165 1787708316.313060 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7d1eb799 (stale) Mirror 4: 0x7d1eb799 (stale) Mirror 5: 0x7d1eb799 (stale) Mirror 6: 0x7d1eb799 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7d1eb799 (stale) Mirror 4: 0x7d1eb799 (stale) Mirror 5: 0x7d1eb799 (stale) Mirror 6: 0x7d1eb799 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000166 1787708316.806982 mapread 0x012b96 thru 0x01fec1 (0xd32c bytes) 000169 1787708316.833953 mapwrite 0x00521d thru 0x00a0c3 (0x4ea7 bytes) 000172 1787708316.886829 write 0x00822a thru 0x008501 (0x2d8 bytes) 000173 1787708316.887814 mapread 0x00c66a thru 0x00d2e8 (0xc7f bytes) 000174 1787708316.901442 mapread 0x000f2d thru 0x001937 (0xa0b bytes) 000175 1787708316.908136 mapwrite 0x03195a thru 0x034466 (0x2b0d bytes) 000176 1787708316.956792 write 0x0085f1 thru 0x013ed1 (0xb8e1 bytes) 000177 1787708316.958151 trunc from 0x040000 to 0x028c95 (0x1736a bytes) 000179 1787708316.976918 mapread 0x0261ca thru 0x028c95 (0x2acc bytes) 000182 1787708316.984866 read 0x005c39 thru 0x00bc4b (0x6013 bytes) 000186 1787708316.993020 MIRROR_RESYNC 5 mirrors 000187 1787708318.394183 MIRROR_RESYNC 5 mirrors 000190 1787708319.102149 MIRROR_EXTEND to 6 mirrors 000191 1787708320.598981 write 0x02af92 thru 0x02bdae (0xe1d bytes) 000192 1787708320.711437 write 0x03cbc6 thru 0x03ffff (0x343a bytes) 000193 1787708320.713284 read 0x014e21 thru 0x01d862 (0x8a42 bytes) 000194 1787708320.729598 mapread 0x021c73 thru 0x02a858 (0x8be6 bytes) 000195 1787708320.761914 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xb0daf139 (stale) Mirror 4: 0xb0daf139 (stale) Mirror 5: 0xb0daf139 (stale) Mirror 7: 0xb0daf139 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xb0daf139 (stale) Mirror 4: 0xb0daf139 (stale) Mirror 5: 0xb0daf139 (stale) Mirror 7: 0xb0daf139 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000196 1787708321.375184 trunc from 0x040000 to 0x013f25 (0x2c0da bytes) 000197 1787708321.408804 mapread 0x00fa83 thru 0x013f25 (0x44a3 bytes) 000198 1787708321.434360 write 0x037580 thru 0x0392b3 (0x1d34 bytes) 000199 1787708321.438467 write 0x0306b8 thru 0x034117 (0x3a60 bytes) 000200 1787708321.468863 trunc from 0x0392b4 to 0x0243a9 (0x14f0a bytes) 000201 1787708321.506265 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x243aa): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x243aa): Mirror 3: 0x50d5808c (stale) Mirror 4: 0x50d5808c (stale) Mirror 5: 0x50d5808c (stale) Mirror 7: 0x50d5808c (stale) CRC-32 checksum value for chunk [0, 0x243aa): Mirror 1: 0x50d5808c Mirror 8: 0x50d5808c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x243aa: skipped 000203 1787708322.795399 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x243aa): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x243aa): Mirror 3: 0x50d5808c (stale) Mirror 4: 0x50d5808c (stale) Mirror 5: 0x50d5808c (stale) Mirror 7: 0x50d5808c (stale) CRC-32 checksum value for chunk [0, 0x243aa): Mirror 1: 0x50d5808c Mirror 8: 0x50d5808c Mirror 9: 0x50d5808c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x243aa: skipped 000206 1787708324.195924 mapread 0x0233af thru 0x0243a9 (0xffb bytes) 000209 1787708324.225199 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x243aa): Mirror 3: 0x50d5808c (stale) Mirror 5: 0x50d5808c (stale) Mirror 7: 0x50d5808c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x243aa): Mirror 3: 0x50d5808c (stale) Mirror 5: 0x50d5808c (stale) Mirror 7: 0x50d5808c (stale) CRC-32 checksum value for chunk [0, 0x243aa): Mirror 1: 0x50d5808c Mirror 8: 0x50d5808c Mirror 9: 0x50d5808c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x243aa: skipped 000210 1787708325.012756 MIRROR_RESYNC 6 mirrors 000211 1787708326.406366 read 0x019fa6 thru 0x01c2d0 (0x232b bytes) 000212 1787708326.417018 read 0x01cb7b thru 0x02044b (0x38d1 bytes) 000213 1787708326.433679 write 0x01ed9d thru 0x022cc9 (0x3f2d bytes) 000214 1787708326.561424 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x243aa): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x243aa): Mirror 3: 0x50d5808c (stale) Mirror 5: 0x50d5808c (stale) Mirror 7: 0x50d5808c (stale) Mirror 8: 0x50d5808c (stale) Mirror 9: 0x50d5808c (stale) CRC-32 checksum value for chunk [0, 0x243aa): Mirror 1: 0xc8728965 Mirror 10: 0xc8728965 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x243aa: skipped 000216 1787708328.018669 write 0x0285b2 thru 0x0308f8 (0x8347 bytes) 000219 1787708328.135784 mapread 0x014aff thru 0x023edf (0xf3e1 bytes) 000220 1787708328.202432 mapread 0x008ba0 thru 0x01047d (0x78de bytes) 000221 1787708328.231426 write 0x00f7b3 thru 0x018588 (0x8dd6 bytes) 000223 1787708328.236617 mapread 0x01a43d thru 0x026e99 (0xca5d bytes) 000224 1787708328.240275 mapread 0x013a19 thru 0x01434a (0x932 bytes) 000225 1787708328.245426 trunc from 0x0308f9 to 0x02560b (0xb2ed bytes) 000226 1787708328.283130 MIRROR_RESYNC 7 mirrors truncating to largest ever: 0x3f14b 000227 1787708330.336417 trunc from 0x02560c to 0x03f14a (0x19b3f bytes) 000229 1787708330.449312 MIRROR_RESYNC 7 mirrors 000231 1787708332.153119 write 0x00c1f7 thru 0x017631 (0xb43b bytes) 000233 1787708332.233717 MIRROR_RESYNC 7 mirrors 000234 1787708333.984762 read 0x001df7 thru 0x008bd4 (0x6dde bytes) 000236 1787708333.993277 read 0x02743b thru 0x031ac4 (0xa68a bytes) 000237 1787708334.002688 read 0x0394b3 thru 0x03f14a (0x5c98 bytes) 000240 1787708334.006272 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) 000241 1787708334.455187 read 0x0216a9 thru 0x02abd4 (0x952c bytes) 000243 1787708334.480127 mapwrite 0x0364c9 thru 0x038e79 (0x29b1 bytes) 000248 1787708334.692244 read 0x0202d3 thru 0x02e4c4 (0xe1f2 bytes) 000249 1787708334.718329 trunc from 0x03f14b to 0x002b64 (0x3c5e6 bytes) 000250 1787708334.779658 write 0x0152ad thru 0x01c2ad (0x7001 bytes) 000251 1787708334.780944 read 0x00430f thru 0x00dba3 (0x9895 bytes) 000253 1787708334.797712 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c2ae): Mirror 3: 0xcd859ec2 (stale) Mirror 5: 0xcd859ec2 (stale) Mirror 7: 0xcd859ec2 (stale) Mirror 10: 0xcd859ec2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c2ae): Mirror 3: 0xcd859ec2 (stale) Mirror 5: 0xcd859ec2 (stale) Mirror 7: 0xcd859ec2 (stale) Mirror 10: 0xcd859ec2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c2ae: skipped 000254 1787708335.417797 mapread 0x013ca4 thru 0x015d89 (0x20e6 bytes) 000258 1787708335.468157 mapread 0x005b98 thru 0x011ad4 (0xbf3d bytes) 000259 1787708335.486616 write 0x0300be thru 0x03cbf4 (0xcb37 bytes) 000261 1787708335.488681 read 0x000feb thru 0x00efdb (0xdff1 bytes) 000262 1787708335.503781 mapwrite 0x0235b7 thru 0x02aadb (0x7525 bytes) 000263 1787708335.613886 write 0x024275 thru 0x02772c (0x34b8 bytes) 000265 1787708335.615618 mapread 0x01b89c thru 0x01f6dd (0x3e42 bytes) 000266 1787708335.616527 read 0x0058dd thru 0x008eec (0x3610 bytes) 000267 1787708335.623261 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3cbf5): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cbf5): Mirror 3: 0x15c350c8 (stale) Mirror 5: 0x15c350c8 (stale) Mirror 7: 0x15c350c8 (stale) Mirror 10: 0x15c350c8 (stale) CRC-32 checksum value for chunk [0, 0x3cbf5): Mirror 1: 0x408ebc28 Mirror 11: 0x408ebc28 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cbf5: skipped 000271 1787708337.064650 mapread 0x018710 thru 0x025d8d (0xd67e bytes) 000272 1787708337.099203 mapwrite 0x032de8 thru 0x039115 (0x632e bytes) 000273 1787708337.278458 read 0x034cc2 thru 0x03cbf4 (0x7f33 bytes) 000278 1787708337.295251 mapread 0x026f11 thru 0x0343d7 (0xd4c7 bytes) 000279 1787708337.308048 mapread 0x01fb21 thru 0x02ae9c (0xb37c bytes) 000280 1787708337.310127 mapwrite 0x03023f thru 0x03293a (0x26fc bytes) 000281 1787708337.358111 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cbf5): Mirror 5: 0x15c350c8 (stale) Mirror 7: 0x15c350c8 (stale) Mirror 10: 0x15c350c8 (stale) Mirror 11: 0x408ebc28 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cbf5): Mirror 5: 0x15c350c8 (stale) Mirror 7: 0x15c350c8 (stale) Mirror 10: 0x15c350c8 (stale) Mirror 11: 0x408ebc28 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cbf5: skipped 000283 1787708338.027951 mapread 0x00583b thru 0x00fa17 (0xa1dd bytes) 000285 1787708338.048299 mapread 0x025396 thru 0x02d137 (0x7da2 bytes) 000286 1787708338.061044 mapwrite 0x0262bf thru 0x026a3e (0x780 bytes) 000287 1787708338.103242 mapread 0x012c6d thru 0x018fa4 (0x6338 bytes) 000288 1787708338.112735 mapwrite 0x00c95a thru 0x0161b5 (0x985c bytes) 000291 1787708338.156195 mapwrite 0x021519 thru 0x02ccbe (0xb7a6 bytes) 000292 1787708338.203615 trunc from 0x03cbf5 to 0x02bcf8 (0x10efc bytes) 000293 1787708338.237920 trunc from 0x02bcf9 to 0x0130d4 (0x18c24 bytes) 000294 1787708338.283014 trunc from 0x0130d5 to 0x017e01 (0x4d2d bytes) 000297 1787708338.312080 mapwrite 0x0044ab thru 0x0117ca (0xd320 bytes) 000298 1787708338.425732 mapread 0x011922 thru 0x017e01 (0x64e0 bytes) 000300 1787708338.438370 mapwrite 0x0071ca thru 0x013f3a (0xcd71 bytes) 000301 1787708338.490927 mapwrite 0x006649 thru 0x013c41 (0xd5f9 bytes) 000302 1787708338.545084 write 0x029210 thru 0x036337 (0xd128 bytes) 000303 1787708338.555418 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x36338): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36338): Mirror 5: 0x341a6809 (stale) Mirror 7: 0x341a6809 (stale) Mirror 10: 0x341a6809 (stale) Mirror 11: 0x9ebdf6e2 (stale) CRC-32 checksum value for chunk [0, 0x36338): Mirror 1: 0x371f123f Mirror 12: 0x371f123f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36338: skipped 000304 1787708340.062153 mapread 0x0075da thru 0x010f77 (0x999e bytes) 000305 1787708340.108831 trunc from 0x036338 to 0x00e69e (0x27c99 bytes) 000307 1787708340.248336 write 0x017b87 thru 0x026f0e (0xf388 bytes) 000309 1787708340.267860 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x26f0f): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26f0f): Mirror 5: 0x67b166f5 (stale) Mirror 7: 0x67b166f5 (stale) Mirror 10: 0x67b166f5 (stale) Mirror 11: 0x17dc9ccc (stale) Mirror 12: 0xbb754383 (stale) CRC-32 checksum value for chunk [0, 0x26f0f): Mirror 1: 0x40bba149 Mirror 13: 0x40bba149 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26f0f: skipped 000310 1787708341.607075 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26f0f): Mirror 5: 0x67b166f5 (stale) Mirror 7: 0x67b166f5 (stale) Mirror 10: 0x67b166f5 (stale) Mirror 11: 0x17dc9ccc (stale) Mirror 12: 0xbb754383 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26f0f): Mirror 5: 0x67b166f5 (stale) Mirror 7: 0x67b166f5 (stale) Mirror 10: 0x67b166f5 (stale) Mirror 11: 0x17dc9ccc (stale) Mirror 12: 0xbb754383 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26f0f: skipped 000311 1787708342.016695 mapread 0x023d1e thru 0x026f0e (0x31f1 bytes) 000312 1787708342.051666 MIRROR_RESYNC 6 mirrors 000314 1787708344.156793 trunc from 0x026f0f to 0x034f48 (0xe03a bytes) 000318 1787708344.315067 mapread 0x02640f thru 0x02848c (0x207e bytes) 000321 1787708344.357780 write 0x03dc54 thru 0x03ffff (0x23ac bytes) 000323 1787708344.362117 read 0x01987e thru 0x01c3e0 (0x2b63 bytes) 000327 1787708344.379793 trunc from 0x040000 to 0x028a15 (0x175ea bytes) 000328 1787708344.408796 read 0x003779 thru 0x00f6ef (0xbf77 bytes) 000331 1787708344.426885 trunc from 0x028a16 to 0x03caca (0x140b5 bytes) 000334 1787708344.449569 write 0x01a009 thru 0x01a3c0 (0x3b8 bytes) 000336 1787708344.452497 write 0x001fbf thru 0x009f27 (0x7f69 bytes) 000337 1787708344.456222 trunc from 0x03cacb to 0x03ee82 (0x23b8 bytes) 000338 1787708344.483466 mapwrite 0x0145d8 thru 0x01c7a0 (0x81c9 bytes) 000339 1787708344.568916 trunc from 0x03ee83 to 0x008e61 (0x36021 bytes) 000340 1787708344.591510 mapread 0x00772d thru 0x008e61 (0x1735 bytes) 000341 1787708344.592673 read 0x003c1d thru 0x008e61 (0x5245 bytes) 000342 1787708344.594781 read 0x0062a9 thru 0x008e61 (0x2bb9 bytes) 000343 1787708344.596221 trunc from 0x008e62 to 0x038f31 (0x300d0 bytes) 000348 1787708344.628587 mapread 0x0100ed thru 0x01044b (0x35f bytes) 000350 1787708344.651722 read 0x018ad5 thru 0x01ccb7 (0x41e3 bytes) 000353 1787708344.658210 mapwrite 0x035e6f thru 0x03ffff (0xa191 bytes) 000354 1787708344.807671 trunc from 0x040000 to 0x02a13f (0x15ec0 bytes) 000355 1787708344.839428 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2a140): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a140): Mirror 5: 0x329159a1 (stale) Mirror 7: 0x329159a1 (stale) Mirror 10: 0x329159a1 (stale) Mirror 11: 0x329159a1 (stale) Mirror 12: 0x329159a1 (stale) CRC-32 checksum value for chunk [0, 0x2a140): Mirror 1: 0x8412b83 Mirror 13: 0x8412b83 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a140: skipped 000356 1787708346.718098 read 0x01196f thru 0x01aaa0 (0x9132 bytes) 000357 1787708346.739672 trunc from 0x02a140 to 0x004930 (0x2580f bytes) 000358 1787708346.876887 write 0x00a461 thru 0x0191a6 (0xed46 bytes) 000359 1787708346.878976 write 0x035263 thru 0x03962f (0x43cd bytes) 000360 1787708346.881760 read 0x008a02 thru 0x00d7db (0x4dda bytes) 000362 1787708346.894707 mapwrite 0x0145c1 thru 0x01be95 (0x78d5 bytes) 000363 1787708347.003025 trunc from 0x039630 to 0x01def1 (0x1b73e bytes) 000364 1787708347.038950 MIRROR_RESYNC 7 mirrors 000367 1787708348.960172 mapwrite 0x017fb7 thru 0x01be8f (0x3ed9 bytes) 000368 1787708349.169020 mapread 0x000fa9 thru 0x00b60a (0xa662 bytes) 000371 1787708349.185900 write 0x016da9 thru 0x01e8e1 (0x7b39 bytes) 000376 1787708349.210067 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e8e2): Mirror 5: 0x4477dd8b (stale) Mirror 10: 0x4477dd8b (stale) Mirror 11: 0x4477dd8b (stale) Mirror 12: 0x4477dd8b (stale) Mirror 13: 0x4477dd8b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e8e2): Mirror 5: 0x4477dd8b (stale) Mirror 10: 0x4477dd8b (stale) Mirror 11: 0x4477dd8b (stale) Mirror 12: 0x4477dd8b (stale) Mirror 13: 0x4477dd8b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e8e2: skipped 000378 1787708349.922654 read 0x003676 thru 0x00f26e (0xbbf9 bytes) 000382 1787708349.934913 mapwrite 0x00a58c thru 0x0186ff (0xe174 bytes) 000383 1787708350.089075 trunc from 0x01e8e2 to 0x02970a (0xae29 bytes) 000385 1787708350.114288 MIRROR_RESYNC 6 mirrors 000389 1787708351.699329 trunc from 0x02970b to 0x0295b5 (0x155 bytes) 000390 1787708351.785154 mapwrite 0x0229fc thru 0x027789 (0x4d8e bytes) 000391 1787708351.842036 mapwrite 0x00f456 thru 0x01d96a (0xe515 bytes) 000393 1787708351.905470 read 0x008c1b thru 0x01488f (0xbc75 bytes) 000395 1787708351.906714 read 0x0179af thru 0x01c0fb (0x474d bytes) 000396 1787708351.908489 trunc from 0x0295b6 to 0x000d62 (0x28853 bytes) 000397 1787708351.941197 MIRROR_RESYNC 6 mirrors 000398 1787708353.216496 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) 000399 1787708353.729129 read 0x000ad0 thru 0x000d62 (0x293 bytes) 000400 1787708353.744855 write 0x02b611 thru 0x0304b2 (0x4ea2 bytes) 000401 1787708353.829481 mapread 0x0111c3 thru 0x017d42 (0x6b80 bytes) 000402 1787708353.852636 read 0x01d67d thru 0x0292e1 (0xbc65 bytes) 000404 1787708353.870217 MIRROR_RESYNC 5 mirrors 000405 1787708355.672346 write 0x00ac20 thru 0x010801 (0x5be2 bytes) 000406 1787708355.872630 mapwrite 0x036e9d thru 0x03a993 (0x3af7 bytes) 000407 1787708356.034397 read 0x019a55 thru 0x026f11 (0xd4bd bytes) 000409 1787708356.050657 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3a994): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a994): Mirror 10: 0x84892863 (stale) Mirror 11: 0x84892863 (stale) Mirror 12: 0x84892863 (stale) Mirror 13: 0x84892863 (stale) CRC-32 checksum value for chunk [0, 0x3a994): Mirror 1: 0x7f9132f0 Mirror 14: 0x7f9132f0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a994: skipped 000415 1787708357.525405 trunc from 0x03a994 to 0x00b3f3 (0x2f5a0 bytes) 000416 1787708357.628773 trunc from 0x00b3f4 to 0x017448 (0xc055 bytes) 000418 1787708357.680869 mapwrite 0x00c6c2 thru 0x01c392 (0xfcd1 bytes) 000419 1787708357.843929 write 0x001f5e thru 0x0115b7 (0xf65a bytes) 000421 1787708357.859066 read 0x01529f thru 0x01a934 (0x5696 bytes) 000422 1787708357.861800 read 0x00f97e thru 0x01bc8a (0xc30d bytes) 000427 1787708357.866674 mapwrite 0x02a233 thru 0x034cc7 (0xaa95 bytes) 000429 1787708358.042626 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x34cc8): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34cc8): Mirror 10: 0xf1e9ea9f (stale) Mirror 11: 0xf1e9ea9f (stale) Mirror 12: 0xf1e9ea9f (stale) Mirror 13: 0xf1e9ea9f (stale) Mirror 14: 0xb2318503 (stale) CRC-32 checksum value for chunk [0, 0x34cc8): Mirror 1: 0x8d4bd030 Mirror 15: 0x8d4bd030 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34cc8: skipped 000430 1787708359.810905 trunc from 0x034cc8 to 0x01483f (0x20488 bytes) 000431 1787708359.923894 write 0x03a4ec thru 0x03ffff (0x5b14 bytes) 000433 1787708359.925530 trunc from 0x040000 to 0x03940a (0x6bf5 bytes) 000434 1787708359.953586 mapwrite 0x01835b thru 0x025204 (0xceaa bytes) 000435 1787708360.098472 mapwrite 0x024c63 thru 0x024e5d (0x1fb bytes) 000436 1787708360.194720 read 0x029433 thru 0x036435 (0xd003 bytes) 000437 1787708360.228975 write 0x016678 thru 0x021846 (0xb1cf bytes) 000438 1787708360.238115 mapwrite 0x0324d7 thru 0x034c17 (0x2741 bytes) 000439 1787708360.309950 mapwrite 0x02d7cb thru 0x039e58 (0xc68e bytes) 000441 1787708360.538871 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39e59): Mirror 10: 0x5ed8eb1 (stale) Mirror 11: 0x5ed8eb1 (stale) Mirror 12: 0x5ed8eb1 (stale) Mirror 13: 0x5ed8eb1 (stale) Mirror 15: 0xbe7a183f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39e59): Mirror 10: 0x5ed8eb1 (stale) Mirror 11: 0x5ed8eb1 (stale) Mirror 12: 0x5ed8eb1 (stale) Mirror 13: 0x5ed8eb1 (stale) Mirror 15: 0xbe7a183f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39e59: skipped 000443 1787708361.023020 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39e59): Mirror 10: 0x5ed8eb1 (stale) Mirror 11: 0x5ed8eb1 (stale) Mirror 12: 0x5ed8eb1 (stale) Mirror 13: 0x5ed8eb1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39e59): Mirror 10: 0x5ed8eb1 (stale) Mirror 11: 0x5ed8eb1 (stale) Mirror 12: 0x5ed8eb1 (stale) Mirror 13: 0x5ed8eb1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39e59: skipped 000446 1787708361.371322 mapwrite 0x0147c1 thru 0x014d14 (0x554 bytes) 000448 1787708361.450369 mapwrite 0x03d321 thru 0x03ffff (0x2cdf bytes) 000449 1787708361.547427 read 0x018cf1 thru 0x01ed03 (0x6013 bytes) 000450 1787708361.564879 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x3aca8ed4 (stale) Mirror 12: 0x3aca8ed4 (stale) Mirror 13: 0x3aca8ed4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x3aca8ed4 (stale) Mirror 12: 0x3aca8ed4 (stale) Mirror 13: 0x3aca8ed4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000451 1787708362.067360 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x3aca8ed4 (stale) Mirror 13: 0x3aca8ed4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x3aca8ed4 (stale) Mirror 13: 0x3aca8ed4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000452 1787708362.411243 read 0x0078d0 thru 0x013cfd (0xc42e bytes) 000453 1787708362.421359 mapwrite 0x008678 thru 0x00ee09 (0x6792 bytes) 000454 1787708362.467721 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x3aca8ed4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x3aca8ed4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000455 1787708362.782101 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) 000456 1787708363.061693 trunc from 0x040000 to 0x013c22 (0x2c3dd bytes) 000459 1787708363.120350 mapwrite 0x03f6f0 thru 0x03ffff (0x910 bytes) 000461 1787708363.222278 write 0x03c694 thru 0x03ffff (0x396c bytes) 000462 1787708363.228482 read 0x01d696 thru 0x020ea7 (0x3812 bytes) 000464 1787708363.243126 mapwrite 0x028ec8 thru 0x032701 (0x983a bytes) 000467 1787708363.399737 write 0x0044d8 thru 0x00f4ae (0xafd7 bytes) 000468 1787708363.427784 MIRROR_EXTEND to 2 mirrors 000469 1787708364.808265 write 0x02a850 thru 0x02b0b0 (0x861 bytes) 000470 1787708364.945430 write 0x00573e thru 0x00f02f (0x98f2 bytes) 000471 1787708364.973348 trunc from 0x040000 to 0x002200 (0x3ddff bytes) 000475 1787708364.999982 mapwrite 0x0259d5 thru 0x025cff (0x32b bytes) 000476 1787708365.123644 write 0x034bcc thru 0x038b79 (0x3fae bytes) 000478 1787708365.130634 mapwrite 0x024dde thru 0x031b18 (0xcd3b bytes) 000479 1787708365.203850 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38b7a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38b7a): Mirror 2: 0xc09b460f (stale) CRC-32 checksum value for chunk [0, 0x38b7a): Mirror 1: 0x75b077b6 Mirror 3: 0x75b077b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38b7a: skipped 000481 1787708366.450297 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38b7a): Mirror 2: 0xc09b460f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38b7a): Mirror 2: 0xc09b460f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38b7a: skipped 000486 1787708366.813945 write 0x01422d thru 0x019a58 (0x582c bytes) 000487 1787708366.845008 write 0x013f60 thru 0x016d09 (0x2daa bytes) 000488 1787708366.850756 mapread 0x0170a5 thru 0x01849d (0x13f9 bytes) 000489 1787708366.858269 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38b7a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38b7a): Mirror 2: 0xc09b460f (stale) CRC-32 checksum value for chunk [0, 0x38b7a): Mirror 1: 0x6620cbdb Mirror 3: 0x6620cbdb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38b7a: skipped 000490 1787708368.022861 mapwrite 0x03441f thru 0x036ef7 (0x2ad9 bytes) 000491 1787708368.233007 mapread 0x02ea05 thru 0x0348d1 (0x5ecd bytes) 000494 1787708368.240117 mapread 0x00520c thru 0x014d43 (0xfb38 bytes) 000495 1787708368.286479 mapread 0x037385 thru 0x038b79 (0x17f5 bytes) 000496 1787708368.288702 mapread 0x01a7a5 thru 0x01dc93 (0x34ef bytes) 000497 1787708368.310678 write 0x004caf thru 0x0117d7 (0xcb29 bytes) 000498 1787708368.315454 mapwrite 0x008aa7 thru 0x00a08c (0x15e6 bytes) 000500 1787708368.372193 mapread 0x0082a8 thru 0x00d751 (0x54aa bytes) 000504 1787708368.375730 read 0x031f69 thru 0x038b79 (0x6c11 bytes) 000505 1787708368.376819 read 0x02afff thru 0x0370f6 (0xc0f8 bytes) 000506 1787708368.386735 trunc from 0x038b7a to 0x005f4b (0x32c2e bytes) 000510 1787708368.441867 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x5f4c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5f4c): Mirror 2: 0x54457036 (stale) Mirror 3: 0x4d49ea31 (stale) CRC-32 checksum value for chunk [0, 0x5f4c): Mirror 1: 0x9c8cc015 Mirror 4: 0x9c8cc015 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5f4c: skipped 000512 1787708369.381773 mapwrite 0x001d6b thru 0x00784b (0x5ae1 bytes) 000513 1787708369.635342 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x784c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x784c): Mirror 2: 0x6bb0696f (stale) Mirror 3: 0x9239d2aa (stale) Mirror 4: 0x13697d48 (stale) CRC-32 checksum value for chunk [0, 0x784c): Mirror 1: 0xdf28e0cc Mirror 5: 0xdf28e0cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x784c: skipped 000514 1787708370.969116 mapwrite 0x030607 thru 0x03ac6b (0xa665 bytes) 000515 1787708371.201408 write 0x0322aa thru 0x03632c (0x4083 bytes) 000516 1787708371.203432 mapwrite 0x01ec00 thru 0x023098 (0x4499 bytes) 000517 1787708371.302394 trunc from 0x03ac6c to 0x0235d2 (0x17699 bytes) 000519 1787708371.348520 write 0x006c78 thru 0x00c992 (0x5d1b bytes) 000520 1787708371.366294 mapread 0x0014a3 thru 0x0046eb (0x3249 bytes) 000521 1787708371.377683 mapwrite 0x020c22 thru 0x028f48 (0x8327 bytes) 000522 1787708371.466979 write 0x023ccd thru 0x0283e7 (0x471b bytes) 000524 1787708371.468503 read 0x01f8d4 thru 0x023252 (0x397f bytes) 000525 1787708371.470228 mapread 0x022d50 thru 0x028a13 (0x5cc4 bytes) 000526 1787708371.471406 trunc from 0x028f49 to 0x0275bf (0x1989 bytes) 000527 1787708371.494594 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x275c0): Mirror 2: 0xce522b38 (stale) Mirror 3: 0x522df9fb (stale) Mirror 4: 0x5f003d59 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x275c0): Mirror 2: 0xce522b38 (stale) Mirror 3: 0x522df9fb (stale) Mirror 4: 0x5f003d59 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x275c0: skipped 000529 1787708371.958440 read 0x013579 thru 0x013f5c (0x9e4 bytes) 000530 1787708371.964129 read 0x020086 thru 0x0275bf (0x753a bytes) 000531 1787708371.976206 mapwrite 0x02d854 thru 0x035998 (0x8145 bytes) 000533 1787708372.058385 mapwrite 0x02fafe thru 0x034dd0 (0x52d3 bytes) 000535 1787708372.123422 read 0x021d0b thru 0x02a391 (0x8687 bytes) 000540 1787708372.133521 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x35999): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35999): Mirror 2: 0xdad63ad3 (stale) Mirror 3: 0x781477de (stale) Mirror 4: 0x24ce1b4c (stale) CRC-32 checksum value for chunk [0, 0x35999): Mirror 1: 0x997b2ad0 Mirror 5: 0x997b2ad0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35999: skipped 000542 1787708373.475418 mapread 0x00cbbd thru 0x014d46 (0x818a bytes) 000546 1787708373.545353 mapread 0x016dbe thru 0x01c349 (0x558c bytes) 000547 1787708373.581495 write 0x0193b3 thru 0x022e8e (0x9adc bytes) 000548 1787708373.740516 write 0x00774e thru 0x00af60 (0x3813 bytes) 000549 1787708373.762357 trunc from 0x035999 to 0x0010f6 (0x348a2 bytes) 000550 1787708373.872263 mapwrite 0x0059ad thru 0x00b7c5 (0x5e19 bytes) 000552 1787708374.066006 write 0x0286b2 thru 0x0317ce (0x911d bytes) 000553 1787708374.074629 MIRROR_RESYNC 5 mirrors 000555 1787708376.275119 trunc from 0x0317cf to 0x00807d (0x29751 bytes) 000556 1787708376.399763 mapread 0x0049ed thru 0x00807d (0x3691 bytes) 000559 1787708376.438675 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x807e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x807e): Mirror 2: 0x6f671895 (stale) Mirror 3: 0x6f671895 (stale) Mirror 4: 0x6f671895 (stale) Mirror 5: 0x6f671895 (stale) CRC-32 checksum value for chunk [0, 0x807e): Mirror 1: 0x6f671895 Mirror 6: 0x6f671895 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x807e: skipped 000560 1787708377.606180 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x807e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x807e): Mirror 2: 0x6f671895 (stale) Mirror 3: 0x6f671895 (stale) Mirror 4: 0x6f671895 (stale) Mirror 5: 0x6f671895 (stale) CRC-32 checksum value for chunk [0, 0x807e): Mirror 1: 0x6f671895 Mirror 6: 0x6f671895 Mirror 7: 0x6f671895 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x807e: skipped 000561 1787708378.822349 trunc from 0x00807e to 0x006806 (0x1877 bytes) 000562 1787708378.934642 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6807): Mirror 2: 0x8be8d4ed (stale) Mirror 3: 0x8be8d4ed (stale) Mirror 4: 0x8be8d4ed (stale) Mirror 5: 0x8be8d4ed (stale) Mirror 7: 0x8be8d4ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6807): Mirror 2: 0x8be8d4ed (stale) Mirror 3: 0x8be8d4ed (stale) Mirror 4: 0x8be8d4ed (stale) Mirror 5: 0x8be8d4ed (stale) Mirror 7: 0x8be8d4ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6807: skipped 000563 1787708379.310978 mapread 0x00062e thru 0x004803 (0x41d6 bytes) 000564 1787708379.337187 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x6807): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6807): Mirror 2: 0x8be8d4ed (stale) Mirror 3: 0x8be8d4ed (stale) Mirror 4: 0x8be8d4ed (stale) Mirror 5: 0x8be8d4ed (stale) Mirror 7: 0x8be8d4ed (stale) CRC-32 checksum value for chunk [0, 0x6807): Mirror 1: 0x8be8d4ed Mirror 8: 0x8be8d4ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6807: skipped 000569 1787708380.730622 mapread 0x004866 thru 0x006806 (0x1fa1 bytes) 000570 1787708380.747833 mapwrite 0x03af5d thru 0x03ffff (0x50a3 bytes) 000572 1787708380.986227 write 0x01dded thru 0x01fa51 (0x1c65 bytes) 000573 1787708380.999606 write 0x02b5dd thru 0x03515a (0x9b7e bytes) 000576 1787708381.009275 read 0x03bb59 thru 0x03ffff (0x44a7 bytes) 000577 1787708381.010308 mapread 0x00d2e4 thru 0x00db8c (0x8a9 bytes) 000578 1787708381.018524 read 0x0140de thru 0x0226c5 (0xe5e8 bytes) 000579 1787708381.029245 mapwrite 0x036eb4 thru 0x0394d5 (0x2622 bytes) 000583 1787708381.088121 mapwrite 0x0349a0 thru 0x03ffff (0xb660 bytes) 000584 1787708381.150072 mapwrite 0x00bee1 thru 0x00c67a (0x79a bytes) 000585 1787708381.191392 trunc from 0x040000 to 0x00a467 (0x35b98 bytes) 000586 1787708381.212882 trunc from 0x00a468 to 0x00b313 (0xeac bytes) 000587 1787708381.233022 mapwrite 0x02c088 thru 0x037217 (0xb190 bytes) 000588 1787708381.543709 write 0x0071c9 thru 0x00cfe3 (0x5e1b bytes) 000591 1787708381.553426 write 0x036c8c thru 0x03b0ed (0x4462 bytes) 000593 1787708381.556257 trunc from 0x03b0ee to 0x00b9b0 (0x2f73d bytes) 000599 1787708381.601524 write 0x00d630 thru 0x01b4c1 (0xde92 bytes) 000600 1787708381.605210 read 0x00c297 thru 0x0103cc (0x4136 bytes) 000602 1787708381.620707 read 0x015904 thru 0x0197fb (0x3ef8 bytes) 000603 1787708381.621325 read 0x008b51 thru 0x010368 (0x7818 bytes) 000605 1787708381.627310 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b4c2): Mirror 2: 0x6e68da5d (stale) Mirror 3: 0x6e68da5d (stale) Mirror 4: 0x6e68da5d (stale) Mirror 5: 0x6e68da5d (stale) Mirror 7: 0x89fa90a1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b4c2): Mirror 2: 0x6e68da5d (stale) Mirror 3: 0x6e68da5d (stale) Mirror 4: 0x6e68da5d (stale) Mirror 5: 0x6e68da5d (stale) Mirror 7: 0x89fa90a1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b4c2: skipped 000606 1787708382.105137 write 0x029e1f thru 0x02a54b (0x72d bytes) 000607 1787708382.111376 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2a54c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a54c): Mirror 2: 0xab91f253 (stale) Mirror 3: 0xab91f253 (stale) Mirror 4: 0xab91f253 (stale) Mirror 5: 0xab91f253 (stale) Mirror 7: 0x5d41627e (stale) CRC-32 checksum value for chunk [0, 0x2a54c): Mirror 1: 0xe26f587f Mirror 8: 0xe26f587f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a54c: skipped 000611 1787708383.430647 mapwrite 0x02ff81 thru 0x037219 (0x7299 bytes) 000612 1787708383.632289 mapwrite 0x021ca1 thru 0x02f4f4 (0xd854 bytes) 000613 1787708383.718931 mapwrite 0x03d77e thru 0x03f4b4 (0x1d37 bytes) 000615 1787708383.793484 write 0x02f609 thru 0x036f69 (0x7961 bytes) 000616 1787708383.796297 trunc from 0x03f4b5 to 0x0074f5 (0x37fbf bytes) 000617 1787708383.828075 mapread 0x001dc7 thru 0x0074f5 (0x572f bytes) 000620 1787708383.850631 trunc from 0x0074f6 to 0x026078 (0x1eb83 bytes) 000626 1787708383.880693 trunc from 0x026079 to 0x030e94 (0xae1c bytes) 000627 1787708383.910632 trunc from 0x030e95 to 0x030214 (0xc80 bytes) 000629 1787708383.939981 mapwrite 0x0047e9 thru 0x00d800 (0x9018 bytes) 000631 1787708384.038741 trunc from 0x030215 to 0x032f91 (0x2d7d bytes) 000632 1787708384.055238 trunc from 0x032f92 to 0x02f431 (0x3b60 bytes) 000633 1787708384.080015 MIRROR_RESYNC 7 mirrors 000638 1787708385.595207 trunc from 0x02f432 to 0x0191cc (0x16265 bytes) 000640 1787708385.665879 mapread 0x008b3f thru 0x012200 (0x96c2 bytes) 000641 1787708385.682492 mapread 0x000b0f thru 0x001c14 (0x1106 bytes) 000642 1787708385.688484 mapwrite 0x004b4b thru 0x0122ff (0xd7b5 bytes) 000643 1787708385.743739 MIRROR_RESYNC 7 mirrors 000644 1787708387.322788 MIRROR_EXTEND to 8 mirrors 000645 1787708388.462007 trunc from 0x0191cd to 0x01f11e (0x5f52 bytes) 000646 1787708388.605467 mapwrite 0x02f538 thru 0x03280a (0x32d3 bytes) 000647 1787708388.816749 read 0x028651 thru 0x02a819 (0x21c9 bytes) 000648 1787708388.835596 MIRROR_RESYNC 8 mirrors 000649 1787708391.758442 mapread 0x03032d thru 0x03280a (0x24de bytes) 000652 1787708391.807907 mapwrite 0x03a6d1 thru 0x03f9e4 (0x5314 bytes) 000653 1787708391.954456 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3f9e5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f9e5): Mirror 2: 0xbc6ad037 (stale) Mirror 3: 0xbc6ad037 (stale) Mirror 4: 0xbc6ad037 (stale) Mirror 5: 0xbc6ad037 (stale) Mirror 7: 0xbc6ad037 (stale) Mirror 8: 0xbc6ad037 (stale) Mirror 9: 0xbc6ad037 (stale) CRC-32 checksum value for chunk [0, 0x3f9e5): Mirror 1: 0x1fb88bd1 Mirror 10: 0x1fb88bd1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f9e5: skipped 000655 1787708393.224983 mapwrite 0x009ffa thru 0x00d1e6 (0x31ed bytes) 000659 1787708393.368323 read 0x02f8a8 thru 0x0363d2 (0x6b2b bytes) 000661 1787708393.380648 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f9e5): Mirror 3: 0xbc6ad037 (stale) Mirror 4: 0xbc6ad037 (stale) Mirror 5: 0xbc6ad037 (stale) Mirror 7: 0xbc6ad037 (stale) Mirror 8: 0xbc6ad037 (stale) Mirror 9: 0xbc6ad037 (stale) Mirror 10: 0x1fb88bd1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f9e5): Mirror 3: 0xbc6ad037 (stale) Mirror 4: 0xbc6ad037 (stale) Mirror 5: 0xbc6ad037 (stale) Mirror 7: 0xbc6ad037 (stale) Mirror 8: 0xbc6ad037 (stale) Mirror 9: 0xbc6ad037 (stale) Mirror 10: 0x1fb88bd1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f9e5: skipped 000662 1787708393.841707 mapwrite 0x027938 thru 0x033543 (0xbc0c bytes) 000663 1787708393.924160 mapwrite 0x032b53 thru 0x03e643 (0xbaf1 bytes) 000665 1787708394.100732 mapread 0x002d41 thru 0x005827 (0x2ae7 bytes) 000666 1787708394.115462 mapread 0x02c440 thru 0x03bd48 (0xf909 bytes) 000668 1787708394.117657 mapwrite 0x00ff01 thru 0x016801 (0x6901 bytes) 000669 1787708394.305442 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f9e5): Mirror 3: 0xbc6ad037 (stale) Mirror 4: 0xbc6ad037 (stale) Mirror 5: 0xbc6ad037 (stale) Mirror 7: 0xbc6ad037 (stale) Mirror 9: 0xbc6ad037 (stale) Mirror 10: 0x1fb88bd1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f9e5): Mirror 3: 0xbc6ad037 (stale) Mirror 4: 0xbc6ad037 (stale) Mirror 5: 0xbc6ad037 (stale) Mirror 7: 0xbc6ad037 (stale) Mirror 9: 0xbc6ad037 (stale) Mirror 10: 0x1fb88bd1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f9e5: skipped 000670 1787708394.719251 trunc from 0x03f9e5 to 0x01679b (0x29249 bytes) 000671 1787708394.735438 write 0x0054c3 thru 0x0111b5 (0xbcf3 bytes) 000672 1787708394.745747 mapread 0x00c24d thru 0x00c3a2 (0x156 bytes) 000675 1787708394.747674 mapwrite 0x01f6ca thru 0x027ba3 (0x84da bytes) 000676 1787708394.826930 read 0x01b185 thru 0x01f8cc (0x4748 bytes) 000677 1787708394.827653 mapread 0x00daaf thru 0x01b7a3 (0xdcf5 bytes) 000678 1787708394.841479 MIRROR_RESYNC 7 mirrors 000680 1787708397.116241 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) 000682 1787708397.572360 mapread 0x006a7f thru 0x012805 (0xbd87 bytes) 000688 1787708397.632075 mapwrite 0x0119d1 thru 0x014e99 (0x34c9 bytes) 000689 1787708397.846567 write 0x003ceb thru 0x0065a3 (0x28b9 bytes) 000690 1787708397.873194 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x27ba4): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27ba4): Mirror 3: 0xa02b70 (stale) Mirror 4: 0xa02b70 (stale) Mirror 7: 0xa02b70 (stale) Mirror 9: 0xa02b70 (stale) Mirror 10: 0xa02b70 (stale) CRC-32 checksum value for chunk [0, 0x27ba4): Mirror 1: 0x7f7a6494 Mirror 11: 0x7f7a6494 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27ba4: skipped 000691 1787708399.113338 mapwrite 0x01e376 thru 0x028f7c (0xac07 bytes) 000692 1787708399.260638 MIRROR_RESYNC 7 mirrors 000693 1787708401.227795 mapread 0x00e636 thru 0x00e9c8 (0x393 bytes) 000696 1787708401.245817 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) 000698 1787708401.870716 read 0x01fab1 thru 0x026d55 (0x72a5 bytes) 000699 1787708401.886175 MIRROR_RESYNC 6 mirrors 000700 1787708402.571293 mapwrite 0x00355f thru 0x011861 (0xe303 bytes) 000702 1787708402.763233 mapread 0x02653e thru 0x028f7c (0x2a3f bytes) 000703 1787708402.774127 mapwrite 0x013b5c thru 0x016130 (0x25d5 bytes) 000708 1787708402.818989 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28f7d): Mirror 3: 0x43848d98 (stale) Mirror 4: 0x43848d98 (stale) Mirror 9: 0x43848d98 (stale) Mirror 11: 0x43848d98 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28f7d): Mirror 3: 0x43848d98 (stale) Mirror 4: 0x43848d98 (stale) Mirror 9: 0x43848d98 (stale) Mirror 11: 0x43848d98 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28f7d: skipped 000709 1787708403.308390 mapread 0x018cfe thru 0x02770f (0xea12 bytes) 000711 1787708403.337468 write 0x001d07 thru 0x00366d (0x1967 bytes) 000712 1787708403.346756 trunc from 0x028f7d to 0x017439 (0x11b43 bytes) 000713 1787708403.381690 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1743a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1743a): Mirror 3: 0xaa8fd4d1 (stale) Mirror 4: 0xaa8fd4d1 (stale) Mirror 9: 0xaa8fd4d1 (stale) Mirror 11: 0xaa8fd4d1 (stale) CRC-32 checksum value for chunk [0, 0x1743a): Mirror 1: 0xa03a4654 Mirror 12: 0xa03a4654 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1743a: skipped 000714 1787708404.551708 read 0x014513 thru 0x015448 (0xf36 bytes) 000715 1787708404.576328 trunc from 0x01743a to 0x01fd20 (0x88e7 bytes) 000718 1787708404.795429 trunc from 0x01fd21 to 0x01bf55 (0x3dcb bytes) 000720 1787708404.843695 write 0x00ceee thru 0x00fbb8 (0x2ccb bytes) 000723 1787708404.859298 read 0x0164d4 thru 0x01a105 (0x3c32 bytes) 000727 1787708404.887074 read 0x0074a0 thru 0x00cb8f (0x56f0 bytes) 000728 1787708404.896935 MIRROR_RESYNC 6 mirrors 000731 1787708406.547949 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) 000736 1787708406.902964 mapread 0x003fe1 thru 0x004381 (0x3a1 bytes) 000737 1787708406.916447 write 0x01d90d thru 0x023040 (0x5734 bytes) 000738 1787708407.001742 trunc from 0x023041 to 0x0049d9 (0x1e667 bytes) 000740 1787708407.032765 trunc from 0x0049da to 0x01d56b (0x18b92 bytes) 000741 1787708407.047655 trunc from 0x01d56c to 0x019f9c (0x35cf bytes) 000742 1787708407.068396 mapwrite 0x01ed01 thru 0x0203a1 (0x16a1 bytes) 000743 1787708407.187921 read 0x004465 thru 0x00a42c (0x5fc8 bytes) 000744 1787708407.199686 trunc from 0x0203a2 to 0x02c3dc (0xc03b bytes) 000745 1787708407.223378 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2c3dd): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c3dd): Mirror 3: 0x914cedd3 (stale) Mirror 4: 0x914cedd3 (stale) Mirror 11: 0x914cedd3 (stale) Mirror 12: 0x914cedd3 (stale) CRC-32 checksum value for chunk [0, 0x2c3dd): Mirror 1: 0x7f69cf6c Mirror 13: 0x7f69cf6c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c3dd: skipped 000746 1787708408.673658 mapwrite 0x01e1b4 thru 0x026b71 (0x89be bytes) 000748 1787708408.957045 mapwrite 0x03ec28 thru 0x03f6f6 (0xacf bytes) 000749 1787708409.114577 write 0x02579d thru 0x02f1e8 (0x9a4c bytes) 000752 1787708409.124438 mapread 0x039f72 thru 0x03f6f6 (0x5785 bytes) 000753 1787708409.125453 mapread 0x03e1e3 thru 0x03e82c (0x64a bytes) 000754 1787708409.126626 trunc from 0x03f6f7 to 0x00b1ac (0x3454a bytes) 000755 1787708409.148244 MIRROR_RESYNC 6 mirrors 000757 1787708410.950087 trunc from 0x00b1ad to 0x0307d5 (0x25629 bytes) 000760 1787708411.107744 mapwrite 0x037e52 thru 0x03a972 (0x2b21 bytes) 000764 1787708411.223149 write 0x025d6c thru 0x030c63 (0xaef8 bytes) 000765 1787708411.236297 mapread 0x02e947 thru 0x036105 (0x77bf bytes) 000766 1787708411.252703 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a973): Mirror 3: 0xed9a2d22 (stale) Mirror 4: 0xed9a2d22 (stale) Mirror 12: 0xed9a2d22 (stale) Mirror 13: 0xed9a2d22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a973): Mirror 3: 0xed9a2d22 (stale) Mirror 4: 0xed9a2d22 (stale) Mirror 12: 0xed9a2d22 (stale) Mirror 13: 0xed9a2d22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a973: skipped 000767 1787708411.943111 mapwrite 0x03f4c2 thru 0x03ffff (0xb3e bytes) 000768 1787708412.094689 write 0x021aec thru 0x024159 (0x266e bytes) 000770 1787708412.119896 trunc from 0x040000 to 0x02bdd5 (0x1422a bytes) 000771 1787708412.143279 read 0x02b7da thru 0x02bdd5 (0x5fc bytes) 000773 1787708412.152185 read 0x02a762 thru 0x02bdd5 (0x1674 bytes) 000775 1787708412.178277 mapwrite 0x00b95b thru 0x017656 (0xbcfc bytes) 000776 1787708412.230055 trunc from 0x02bdd6 to 0x009ea1 (0x21f34 bytes) 000777 1787708412.249515 mapread 0x006b76 thru 0x009794 (0x2c1f bytes) 000780 1787708412.256149 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9ea2): Mirror 4: 0xd306faa3 (stale) Mirror 12: 0xd306faa3 (stale) Mirror 13: 0xd306faa3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9ea2): Mirror 4: 0xd306faa3 (stale) Mirror 12: 0xd306faa3 (stale) Mirror 13: 0xd306faa3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9ea2: skipped 000782 1787708413.027525 read 0x0077ee thru 0x009ea1 (0x26b4 bytes) 000784 1787708413.033878 write 0x015eae thru 0x0164d8 (0x62b bytes) 000785 1787708413.035037 mapwrite 0x01751e thru 0x01bbbb (0x469e bytes) 000786 1787708413.214897 trunc from 0x01bbbc to 0x017893 (0x4328 bytes) 000787 1787708413.264584 trunc from 0x017894 to 0x01db03 (0x6270 bytes) 000789 1787708413.303971 mapread 0x0142c8 thru 0x017b83 (0x38bc bytes) 000792 1787708413.308603 trunc from 0x01db04 to 0x027e67 (0xa364 bytes) 000793 1787708413.341112 trunc from 0x027e68 to 0x0278b9 (0x5ae bytes) 000796 1787708413.367215 MIRROR_RESYNC 4 mirrors 000797 1787708415.276180 write 0x0136f0 thru 0x019107 (0x5a18 bytes) 000798 1787708415.436893 mapwrite 0x03c32a thru 0x03ffff (0x3cd6 bytes) 000799 1787708415.580821 trunc from 0x040000 to 0x02c5f9 (0x13a06 bytes) 000801 1787708415.610514 mapwrite 0x026bdb thru 0x02bb6e (0x4f94 bytes) 000802 1787708415.697294 trunc from 0x02c5fa to 0x024b8d (0x7a6c bytes) 000803 1787708415.723674 mapread 0x01608a thru 0x0216c2 (0xb639 bytes) 000804 1787708415.735045 write 0x033969 thru 0x038250 (0x48e8 bytes) 000805 1787708415.736276 read 0x01bacc thru 0x0221f5 (0x672a bytes) 000806 1787708415.736921 trunc from 0x038251 to 0x0376d6 (0xb7a bytes) 000807 1787708415.769763 trunc from 0x0376d7 to 0x0011ea (0x364ec bytes) 000808 1787708415.809274 MIRROR_RESYNC 4 mirrors 000810 1787708416.672265 read 0x00023d thru 0x0011ea (0xfae bytes) 000811 1787708416.683469 MIRROR_RESYNC 4 mirrors 000812 1787708417.110376 MIRROR_EXTEND to 5 mirrors 000813 1787708418.074801 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) 000814 1787708418.476530 read 0x000f27 thru 0x0011ea (0x2c4 bytes) 000815 1787708418.484776 mapwrite 0x01853b thru 0x02740c (0xeed2 bytes) 000819 1787708418.769493 mapread 0x008a67 thru 0x00e1a1 (0x573b bytes) 000820 1787708418.789078 MIRROR_RESYNC 4 mirrors 000821 1787708420.446509 trunc from 0x02740d to 0x0206de (0x6d2e bytes) 000822 1787708420.570697 read 0x011095 thru 0x020048 (0xefb4 bytes) 000823 1787708420.583610 mapwrite 0x01f2ee thru 0x023b6f (0x4882 bytes) 000824 1787708420.707909 write 0x029861 thru 0x02a95c (0x10fc bytes) 000827 1787708420.711763 trunc from 0x02a95d to 0x033761 (0x8e05 bytes) 000828 1787708420.747254 mapwrite 0x03ee4c thru 0x03ffff (0x11b4 bytes) 000830 1787708420.895201 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x2d007c8e (stale) Mirror 14: 0x2d007c8e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x2d007c8e (stale) Mirror 14: 0x2d007c8e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000831 1787708421.314483 write 0x001070 thru 0x00ca6d (0xb9fe bytes) 000834 1787708421.327088 mapwrite 0x031188 thru 0x033f8b (0x2e04 bytes) 000836 1787708421.410279 write 0x020743 thru 0x02938e (0x8c4c bytes) 000839 1787708421.434524 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x2d007c8e (stale) Mirror 14: 0x2d007c8e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdfcbb818 Mirror 15: 0xdfcbb818 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000840 1787708422.812391 write 0x008a21 thru 0x00aafc (0x20dc bytes) truncating to largest ever: 0x3fc2c 000841 1787708422.961273 trunc from 0x040000 to 0x03fc2b (0x3d4 bytes) 000842 1787708422.987920 mapread 0x0001c3 thru 0x000ffe (0xe3c bytes) 000843 1787708423.003489 MIRROR_RESYNC 4 mirrors 000844 1787708425.027221 trunc from 0x03fc2c to 0x0253ec (0x1a83f bytes) 000846 1787708425.119806 trunc from 0x0253ed to 0x0292a9 (0x3ebd bytes) 000847 1787708425.175795 read 0x027071 thru 0x0292a9 (0x2239 bytes) 000849 1787708425.184774 write 0x01b2f5 thru 0x029caa (0xe9b6 bytes) 000850 1787708425.204093 trunc from 0x029cab to 0x033f28 (0xa27e bytes) 000853 1787708425.229359 read 0x02f01b thru 0x033f28 (0x4f0e bytes) 000854 1787708425.242924 read 0x012e06 thru 0x02175a (0xe955 bytes) 000856 1787708425.244871 trunc from 0x033f29 to 0x036066 (0x213e bytes) 000857 1787708425.268178 mapwrite 0x0051f5 thru 0x00af33 (0x5d3f bytes) 000858 1787708425.352229 mapread 0x017172 thru 0x021660 (0xa4ef bytes) 000859 1787708425.355144 write 0x02b9ec thru 0x039315 (0xd92a bytes) 000860 1787708425.358961 mapwrite 0x0208ef thru 0x0304d8 (0xfbea bytes) 000862 1787708425.430881 MIRROR_RESYNC 4 mirrors 000865 1787708426.963089 mapread 0x02435f thru 0x02a25c (0x5efe bytes) 000868 1787708426.982706 write 0x014c61 thru 0x01a129 (0x54c9 bytes) 000869 1787708427.056896 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39316): Mirror 4: 0x68568869 (stale) Mirror 15: 0x68568869 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39316): Mirror 4: 0x68568869 (stale) Mirror 15: 0x68568869 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39316: skipped 000870 1787708427.438610 mapread 0x021e0b thru 0x026ee6 (0x50dc bytes) 000871 1787708427.466111 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39316): Mirror 4: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39316): Mirror 4: 0x68568869 (stale) Mirror 15: 0x68568869 (stale) CRC-32 checksum value for chunk [0, 0x39316): Mirror 1: 0xf310ce4c Mirror 16: 0xf310ce4c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39316: skipped 000872 1787708428.685108 mapread 0x00c81d thru 0x00d955 (0x1139 bytes) 000873 1787708428.708695 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x39316): Mirror 4: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39316): Mirror 4: 0x68568869 (stale) Mirror 15: 0x68568869 (stale) CRC-32 checksum value for chunk [0, 0x39316): Mirror 1: 0xf310ce4c Mirror 16: 0xf310ce4c Mirror 17: 0xf310ce4c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39316: skipped 000875 1787708430.027671 mapwrite 0x02d447 thru 0x02d683 (0x23d bytes) 000876 1787708430.245152 write 0x002c67 thru 0x004f32 (0x22cc bytes) 000877 1787708430.257689 write 0x01cf1f thru 0x025c5e (0x8d40 bytes) 000878 1787708430.269958 write 0x01cd10 thru 0x02196b (0x4c5c bytes) 000879 1787708430.271936 mapwrite 0x006ea4 thru 0x00e57c (0x76d9 bytes) 000880 1787708430.349610 trunc from 0x039316 to 0x00a260 (0x2f0b5 bytes) 000882 1787708430.379156 write 0x037ff1 thru 0x03cdf4 (0x4e04 bytes) 000883 1787708430.384928 mapwrite 0x00289c thru 0x00ee37 (0xc59c bytes) 000884 1787708430.456811 mapread 0x0220ab thru 0x0293e5 (0x733b bytes) 000887 1787708430.473987 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3cdf5): Mirror 4: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cdf5): Mirror 4: 0x8e18890f (stale) Mirror 15: 0x8e18890f (stale) Mirror 16: 0x895dff12 (stale) Mirror 17: 0x895dff12 (stale) CRC-32 checksum value for chunk [0, 0x3cdf5): Mirror 1: 0xf70de8ef Mirror 18: 0xf70de8ef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cdf5: skipped 000888 1787708431.961799 read 0x0177f9 thru 0x01e3ae (0x6bb6 bytes) 000890 1787708431.971195 read 0x012366 thru 0x016363 (0x3ffe bytes) 000892 1787708431.982379 trunc from 0x03cdf5 to 0x004249 (0x38bab bytes) 000894 1787708432.080738 mapwrite 0x03d9b1 thru 0x03ffff (0x264f bytes) 000895 1787708432.153526 write 0x0284de thru 0x02a693 (0x21b6 bytes) 000896 1787708432.165084 trunc from 0x040000 to 0x029a66 (0x16599 bytes) 000897 1787708432.183810 read 0x005796 thru 0x008968 (0x31d3 bytes) 000898 1787708432.196333 mapread 0x021713 thru 0x029a66 (0x8354 bytes) 000900 1787708432.200369 write 0x0203a1 thru 0x023665 (0x32c5 bytes) 000901 1787708432.202079 write 0x018ef6 thru 0x01c2cf (0x33da bytes) 000904 1787708432.204076 mapwrite 0x01e790 thru 0x022300 (0x3b71 bytes) 000907 1787708432.296115 mapwrite 0x0055db thru 0x007d03 (0x2729 bytes) 000908 1787708432.349970 read 0x01aa07 thru 0x01bb7d (0x1177 bytes) 000909 1787708432.351186 read 0x0191ab thru 0x023b1c (0xa972 bytes) 000911 1787708432.353258 trunc from 0x029a67 to 0x037b35 (0xe0cf bytes) 000912 1787708432.373161 read 0x008835 thru 0x008c40 (0x40c bytes) 000913 1787708432.374503 mapread 0x009839 thru 0x00bd6c (0x2534 bytes) 000914 1787708432.381939 MIRROR_RESYNC 6 mirrors 000917 1787708434.729270 write 0x01c07e thru 0x02566f (0x95f2 bytes) 000919 1787708434.802499 write 0x00e387 thru 0x0163b7 (0x8031 bytes) 000920 1787708434.817217 read 0x000323 thru 0x008313 (0x7ff1 bytes) 000921 1787708434.827024 read 0x01336d thru 0x01ffa4 (0xcc38 bytes) 000923 1787708434.841185 mapwrite 0x03f756 thru 0x03ffff (0x8aa bytes) 000925 1787708434.943454 trunc from 0x040000 to 0x007aa9 (0x38556 bytes) 000926 1787708434.958766 read 0x0047cf thru 0x005f44 (0x1776 bytes) 000929 1787708434.961427 trunc from 0x007aaa to 0x000580 (0x7529 bytes) 000931 1787708434.978894 write 0x01a113 thru 0x025b63 (0xba51 bytes) 000934 1787708434.985984 trunc from 0x025b64 to 0x028704 (0x2ba1 bytes) 000936 1787708435.007173 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28705): Mirror 4: 0xa3649663 (stale) Mirror 15: 0xa3649663 (stale) Mirror 16: 0xa3649663 (stale) Mirror 17: 0xa3649663 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28705): Mirror 4: 0xa3649663 (stale) Mirror 15: 0xa3649663 (stale) Mirror 16: 0xa3649663 (stale) Mirror 17: 0xa3649663 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28705: skipped 000940 1787708435.427093 write 0x02c625 thru 0x039ade (0xd4ba bytes) 000942 1787708435.430260 trunc from 0x039adf to 0x01a075 (0x1fa69 bytes) 000944 1787708435.466056 read 0x0062f6 thru 0x00697c (0x687 bytes) 000945 1787708435.484201 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a076): Mirror 4: 0x5bb2d5b (stale) Mirror 16: 0x5bb2d5b (stale) Mirror 17: 0x5bb2d5b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a076): Mirror 4: 0x5bb2d5b (stale) Mirror 16: 0x5bb2d5b (stale) Mirror 17: 0x5bb2d5b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a076: skipped 000946 1787708435.942666 mapwrite 0x01be19 thru 0x0258af (0x9a97 bytes) 000947 1787708436.082973 mapread 0x01f595 thru 0x0258af (0x631b bytes) 000952 1787708436.092853 trunc from 0x0258b0 to 0x001483 (0x2442c bytes) 000955 1787708436.133098 mapwrite 0x001faf thru 0x00b258 (0x92aa bytes) 000956 1787708436.254343 mapread 0x005fe5 thru 0x00b258 (0x5274 bytes) 000958 1787708436.262437 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xb259): Mirror 4: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb259): Mirror 4: 0xd90b7e4d (stale) Mirror 16: 0xd90b7e4d (stale) Mirror 17: 0xd90b7e4d (stale) CRC-32 checksum value for chunk [0, 0xb259): Mirror 1: 0x41b28742 Mirror 18: 0x41b28742 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb259: skipped 000959 1787708437.292585 trunc from 0x00b259 to 0x016fe5 (0xbd8d bytes) 000961 1787708437.433021 mapread 0x009e2a thru 0x00c557 (0x272e bytes) 000962 1787708437.449393 write 0x00515f thru 0x01109e (0xbf40 bytes) 000965 1787708437.452577 mapwrite 0x02572e thru 0x02b2cc (0x5b9f bytes) 000966 1787708437.524904 mapwrite 0x016884 thru 0x0190b4 (0x2831 bytes) 000967 1787708437.580322 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2b2cd): Mirror 4: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b2cd): Mirror 4: 0x2f0f02c7 (stale) Mirror 16: 0x2f0f02c7 (stale) Mirror 17: 0x2f0f02c7 (stale) Mirror 18: 0x6a896409 (stale) CRC-32 checksum value for chunk [0, 0x2b2cd): Mirror 1: 0xc0ff7010 Mirror 19: 0xc0ff7010 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b2cd: skipped 000968 1787708438.633197 trunc from 0x02b2cd to 0x024e3c (0x6490 bytes) 000969 1787708438.774066 read 0x016b7c thru 0x01713c (0x5c1 bytes) 000972 1787708438.794951 write 0x01d499 thru 0x028cf2 (0xb85a bytes) 000973 1787708438.803204 trunc from 0x028cf3 to 0x02daa4 (0x4db2 bytes) 000974 1787708438.831592 MIRROR_RESYNC 6 mirrors 000977 1787708440.390004 write 0x03750c thru 0x037802 (0x2f7 bytes) 000979 1787708440.521410 write 0x0239c5 thru 0x02c426 (0x8a62 bytes) 000980 1787708440.538209 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37803): Mirror 16: 0x15324d8b (stale) Mirror 17: 0x15324d8b (stale) Mirror 18: 0x15324d8b (stale) Mirror 19: 0x15324d8b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37803): Mirror 16: 0x15324d8b (stale) Mirror 17: 0x15324d8b (stale) Mirror 18: 0x15324d8b (stale) Mirror 19: 0x15324d8b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37803: skipped 000981 1787708441.149078 read 0x012bb7 thru 0x021643 (0xea8d bytes) 000982 1787708441.160686 read 0x012595 thru 0x01b941 (0x93ad bytes) 000983 1787708441.161321 read 0x034315 thru 0x037802 (0x34ee bytes) 000984 1787708441.169733 trunc from 0x037803 to 0x0102c7 (0x2753b bytes) 000986 1787708441.197647 mapread 0x007a03 thru 0x0102c7 (0x88c5 bytes) 000987 1787708441.199096 mapread 0x008406 thru 0x0102c7 (0x7ec2 bytes) 000988 1787708441.200108 mapwrite 0x03bdd0 thru 0x03ffff (0x4230 bytes) 000989 1787708441.281439 read 0x032749 thru 0x03ffff (0xd8b7 bytes) 000991 1787708441.297054 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x3da04043 (stale) Mirror 17: 0x3da04043 (stale) Mirror 18: 0x3da04043 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x3da04043 (stale) Mirror 17: 0x3da04043 (stale) Mirror 18: 0x3da04043 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000996 1787708441.654624 trunc from 0x040000 to 0x01fb35 (0x204ca bytes) 000997 1787708441.668704 trunc from 0x01fb36 to 0x012a77 (0xd0be bytes) 001000 1787708441.683495 read 0x0093d0 thru 0x01052b (0x715c bytes) All operations completed A-OK! PASS 70a (157s) == sanity-flr test 71: check mirror extend parent fid ==== 21:40:47 (1787708447) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 21:40:50 (1787708450) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.258459 s, 8.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144491 s, 7.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 21:40:58 (1787708458) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre2 Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8944160 .. Extending file size to 5164608 .. Extending file size to 8202144 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8421088 .. Extending file size to 9116000 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57465 57466 57467 57469 57470 done 192.168.203.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre2 (opts:) 192.168.203.136@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 21:42:20 (1787708540) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre2 Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1365952 .. Extending file size to 4489728 .. done Extending file size to 2335648 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6530112 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7087904 .. Extending file size to 8938528 .. Extending file size to 8228800 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8463136 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8601376 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9057984 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9311872 .. Waiting 58875 58876 58878 58879 58881 done 192.168.203.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre2 (opts:) 192.168.203.136@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 21:43:42 (1787708622) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0860017 s, 24.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.483215 s, 4.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xeb5a8549 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 21:44:01 (1787708641) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0618827 s, 33.9 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 21:44:09 (1787708649) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f3:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 21:44:18 (1787708658) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.187917 s, 11.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 21:44:26 (1787708666) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.153705 s, 13.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 21:44:34 (1787708674) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.282068 s, 7.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 21:45:07 (1787708707) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg336-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154251 s, 6.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } PASS 204d (57s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 21:46:04 (1787708764) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg336-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.278112 s, 7.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (56s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 21:47:00 (1787708820) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg336-server mds-ost sync done. Creating new pool test_204f oleg336-server: Pool lustre.test_204f created Adding targets to pool oleg336-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg336-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.218443 s, 9.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x305:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x305:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg336-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg336-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg336-server: Pool lustre.test_204f destroyed PASS 204f (74s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 21:48:14 (1787708894) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 21:48:23 (1787708903) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 21:48:32 (1787708912) oleg336-server: Pool lustre.test_206 created Adding targets to pool oleg336-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg336-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg336-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg336-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg336-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 21:48:59 (1787708939) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093835 s, 11.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133934 s, 7.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xd922b4f0 Mirror 3: 0xd922b4f0 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 21:49:12 (1787708952) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 21:49:15 (1787708955) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 21:49:19 (1787708959) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.109638 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x25b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xc50000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xc50000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xc50000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xcf0000 Read this (4 bytes): BBBB 192.168.203.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 21:49:56 (1787708996) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0212501 s, 386 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 21:50:03 (1787709003) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558454 s, 18.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x218 lmm_fid: [0x200000404:0x218:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x218 lmm_fid: [0x200000404:0x218:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834456 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-25 21:50:04.000000000 -0400 Modify: 2026-08-25 21:50:04.000000000 -0400 Change: 2026-08-25 21:50:06.000000000 -0400 Birth: 2026-08-25 21:50:04.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 21:50:11 (1787709011) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 21:50:14 (1787709014) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.427343 s, 95.8 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0681846 s, 60.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 25 21:50 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21a:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 25 21:50 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 21:50:23 (1787709023) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 21:50:31 (1787709031) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1844 sec =========== 21:50:40 (1787709040) === sanity-flr: start cleanup 21:50:42 (1787709042) === === sanity-flr: finish cleanup 21:50:49 (1787709049) ===