-----============= acceptance-small: sanity-flr ============----- Wed Aug 19 07:00:52 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg148-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 07:01:12 (1787137272) === oleg148-client.virtnet: executing check_config_client /mnt/lustre oleg148-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg148-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff889f588fd800.idle_timeout=debug osc.lustre-OST0001-osc-ffff889f588fd800.idle_timeout=debug disable quota as required oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 07:01:48 (1787137308) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7623] == sanity-flr test 0a: lfs mirror create with -N option == 07:01:51 (1787137311) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 07:02:12 (1787137332) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 07:02:16 (1787137336) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 07:02:20 (1787137340) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 07:02:46 (1787137366) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 07:02:49 (1787137369) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 07:02:53 (1787137373) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff889f588fd800.stats=clear osc.lustre-OST0001-osc-ffff889f588fd800.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 07:02:59 (1787137379) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 07:03:09 (1787137389) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 07:03:18 (1787137398) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 07:03:21 (1787137401) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 07:03:29 (1787137409) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 07:03:39 (1787137419) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 07:03:48 (1787137428) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 07:03:52 (1787137432) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 07:04:00 (1787137440) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 07:04:14 (1787137454) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 07:04:21 (1787137461) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 1.64774 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160034 s, 6.6 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 07:04:34 (1787137474) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.635642 s, 33.0 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 07:04:44 (1787137484) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 07:04:56 (1787137496) Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff889f588fd800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff889f588fd800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff889f588fd800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff889f588fd800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg148-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (73s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 07:06:09 (1787137569) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.571879 s, 16.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff889f588fd800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff889f588fd800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 07:06:53 (1787137613) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff889f588fd800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff889f588fd800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff889f588fd800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff889f588fd800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg148-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (84s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 07:08:17 (1787137697) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.17766 s, 10.1 MB/s -rw-r--r-- 1 root root 52428800 Aug 19 07:08 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.107s user 0m0.001s sys 0m0.197s Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff889f588fd800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff889f588fd800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.438s user 0m0.000s sys 0m0.235s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff889f588fd800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff889f588fd800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.372s user 0m0.001s sys 0m0.176s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg148-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg148-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff889f588fd800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff889f588fd800.ost_server_uuid in FULL state after 0 sec PASS 33b (105s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 07:10:02 (1787137802) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 07:10:06 (1787137806) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 07:10:09 (1787137809) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 07:10:13 (1787137813) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 07:10:21 (1787137821) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.533124 s, 23.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 07:10:33 (1787137833) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364324 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399325 s, 26.3 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 07:10:41 (1787137841) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 07:10:51 (1787137851) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00297857 s, 344 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00625996 s, 164 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00255353 s, 401 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00300197 s, 341 kB/s flv=10 olv=10 PASS 36d (13s) == sanity-flr test 37: mirror I/O API verification ======= 07:11:04 (1787137864) create test files with size 15737556 .. 02362957cade34bfc822c2e3d48ae476 - 3611d45f27bef831edea899c99d1bbf3 - 63e9c19969c1ffe79a223de27a8fb099 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 07:11:29 (1787137889) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 07:11:47 (1787137907) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 07:11:54 (1787137914) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198931 s, 5.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027905 s, 37.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcb:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcc:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 07:12:03 (1787137923) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.253535 s, 8.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.502835 s, 8.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.157727 s, 13.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 07:12:22 (1787137942) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 07:12:26 (1787137946) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 07:12:30 (1787137950) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302975 s, 34.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 07:12:38 (1787137958) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 07:12:41 (1787137961) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd1:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd1:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 07:12:55 (1787137975) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.431903 s, 24.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (16s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 07:13:10 (1787137990) llite.lustre-ffff889f49572800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.250898 s, 41.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff889f58930000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 07:13:25 (1787138005) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (44s) == sanity-flr test 46: Verify setstripe --copy option ==== 07:14:10 (1787138050) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (39s) == sanity-flr test 47: Verify mirror obj alloc =========== 07:14:48 (1787138088) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 07:14:53 (1787138093) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.252868 s, 12.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.131079 s, 24.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x32f78289 (nosync,stale) Mirror 2: 0x32f78289 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 564004c362755d6d6538796021b787f6 - mirror 1 checksum: fe74eb70644883d9fe48adba191472a9 - mirror 2 checksum: 564004c362755d6d6538796021b787f6 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-19T07:14:55-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-19T07:14:55-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe7:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 564004c362755d6d6538796021b787f6 - mirror 1 checksum: fe74eb70644883d9fe48adba191472a9 - mirror 2 checksum: fe74eb70644883d9fe48adba191472a9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-19T07:14:55-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-19T07:14:55-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe7:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 07:15:06 (1787138106) Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 07:15:13 (1787138113) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 07:15:38 (1787138138) targets are mounted 07:15:38 (1787138138) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035082 s, 29.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } PASS 50A (51s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 07:15:57 (1787138157) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 07:16:02 (1787138162) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 07:16:05 (1787138165) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00511127 s, 3.2 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 07:16:13 (1787138173) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 07:16:16 (1787138176) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00913759 s, 448 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.139618 s, 29.3 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 07:16:24 (1787138184) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 07:16:49 (1787138209) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 07:17:19 (1787138239) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 07:17:43 (1787138263) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 07:18:09 (1787138289) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 07:18:17 (1787138297) mirror create pid 54331 mirror split pid 54332 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54331 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54332 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 07:19:26 (1787138366) Seed set to 8368 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x150a9 000004 1787138369.093288 trunc from 00000000 to 0x0150a8 (0x150a9 bytes) 000007 1787138369.119067 mapread 0x00d681 thru 0x0150a8 (0x7a28 bytes) 000008 1787138369.173728 read 0x0004e9 thru 0x00337b (0x2e93 bytes) truncating to largest ever: 0x3f7bc 000009 1787138369.182520 trunc from 0x0150a9 to 0x03f7bb (0x2a713 bytes) 000010 1787138369.224052 trunc from 0x03f7bc to 0x024d14 (0x1aaa7 bytes) 000014 1787138369.250678 write 0x0207de thru 0x02fb32 (0xf355 bytes) 000015 1787138369.262189 trunc from 0x02fb33 to 0x024898 (0xb29a bytes) 000019 1787138369.304104 write 0x01fa53 thru 0x02a674 (0xac22 bytes) 000020 1787138369.335662 read 0x00adf3 thru 0x015334 (0xa542 bytes) 000021 1787138369.337455 write 0x01246a thru 0x0183cb (0x5f62 bytes) 000024 1787138369.386441 write 0x03a8e2 thru 0x03ffff (0x571e bytes) 000025 1787138369.388876 write 0x0082aa thru 0x017ab8 (0xf80f bytes) 000027 1787138369.395606 mapread 0x01af93 thru 0x025f33 (0xafa1 bytes) 000028 1787138369.417852 mapwrite 0x029105 thru 0x0312e6 (0x81e2 bytes) 000029 1787138369.531551 trunc from 0x040000 to 0x002984 (0x3d67b bytes) 000031 1787138369.551164 mapread 0x001a36 thru 0x002984 (0xf4f bytes) 000032 1787138369.557452 MIRROR_EXTEND to 2 mirrors 000033 1787138370.661384 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15c:0x0]: Invalid argument (22) 000034 1787138370.819467 read 0x0020cf thru 0x002984 (0x8b6 bytes) 000035 1787138370.831595 read 0x0004ae thru 0x002984 (0x24d7 bytes) 000036 1787138370.840356 trunc from 0x002985 to 0x01d04b (0x1a6c7 bytes) 000037 1787138370.900879 mapread 0x0171e4 thru 0x01a808 (0x3625 bytes) 000044 1787138370.929778 mapwrite 0x03e372 thru 0x03ffff (0x1c8e bytes) 000046 1787138371.027814 trunc from 0x040000 to 0x020655 (0x1f9aa bytes) 000047 1787138371.050219 write 0x0326a4 thru 0x03e636 (0xbf93 bytes) 000049 1787138371.054903 write 0x01e942 thru 0x021211 (0x28d0 bytes) 000050 1787138371.061487 read 0x023552 thru 0x0270b5 (0x3b64 bytes) 000053 1787138371.074492 mapread 0x028b62 thru 0x0328a8 (0x9d47 bytes) 000054 1787138371.076664 read 0x025a7e thru 0x02650c (0xa8f bytes) 000058 1787138371.084674 MIRROR_EXTEND to 2 mirrors 000059 1787138372.165140 read 0x0173f4 thru 0x0259f7 (0xe604 bytes) 000060 1787138372.202188 trunc from 0x03e637 to 0x0235a4 (0x1b092 bytes) 000061 1787138372.417547 read 0x009bfd thru 0x0110c6 (0x74ca bytes) 000062 1787138372.438419 write 0x004897 thru 0x00abc0 (0x632a bytes) 000063 1787138372.440528 read 0x00b9bc thru 0x00e9a5 (0x2fea bytes) 000064 1787138372.441528 trunc from 0x0235a5 to 0x02b115 (0x7b71 bytes) 000065 1787138372.469508 mapread 0x003956 thru 0x004327 (0x9d2 bytes) 000068 1787138372.473793 write 0x01e709 thru 0x0204ae (0x1da6 bytes) 000073 1787138372.478888 read 0x0103d6 thru 0x019414 (0x903f bytes) 000077 1787138372.489320 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15c:0x0]: Invalid argument (22) 000083 1787138372.665676 mapwrite 0x02a178 thru 0x037f1f (0xdda8 bytes) 000085 1787138372.774455 write 0x010c2f thru 0x0140c0 (0x3492 bytes) 000086 1787138372.787459 read 0x02ea18 thru 0x037f1f (0x9508 bytes) 000087 1787138372.788679 trunc from 0x037f20 to 0x02d009 (0xaf16 bytes) 000090 1787138372.808836 write 0x00b63c thru 0x016e6e (0xb833 bytes) 000091 1787138372.822400 trunc from 0x02d00a to 0x0241e6 (0x8e23 bytes) 000092 1787138372.840167 read 0x016ddc thru 0x0189d8 (0x1bfd bytes) 000095 1787138372.852547 write 0x001676 thru 0x00490d (0x3298 bytes) 000096 1787138372.854711 read 0x01a3ea thru 0x0241e6 (0x9dfd bytes) 000099 1787138372.857158 mapwrite 0x0066a0 thru 0x00b65c (0x4fbd bytes) 000101 1787138372.918772 mapread 0x0050cd thru 0x00d763 (0x8697 bytes) 000102 1787138372.943447 MIRROR_EXTEND to 2 mirrors 000105 1787138374.500104 write 0x02c5ec thru 0x02e6a7 (0x20bc bytes) 000107 1787138374.539152 mapwrite 0x03ddbb thru 0x03ded0 (0x116 bytes) 000108 1787138374.641208 trunc from 0x03ded1 to 0x02ca0c (0x114c4 bytes) 000109 1787138374.667450 write 0x01d28b thru 0x022b9d (0x5913 bytes) 000110 1787138374.694595 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15c:0x0]: Invalid argument (22) 000111 1787138375.043988 mapwrite 0x037466 thru 0x03b043 (0x3bde bytes) 000112 1787138375.128610 read 0x038ad1 thru 0x03b043 (0x2573 bytes) 000113 1787138375.129417 mapread 0x01dad0 thru 0x02969c (0xbbcd bytes) 000114 1787138375.156124 write 0x00e6fc thru 0x0159f2 (0x72f7 bytes) 000116 1787138375.173077 MIRROR_EXTEND to 2 mirrors 000117 1787138376.038898 write 0x00205f thru 0x00a3a4 (0x8346 bytes) 000118 1787138376.125196 write 0x002cb0 thru 0x00b47b (0x87cc bytes) 000119 1787138376.134370 mapwrite 0x03bb06 thru 0x03ffff (0x44fa bytes) 000122 1787138376.294724 mapread 0x03eb80 thru 0x03ffff (0x1480 bytes) 000123 1787138376.300673 MIRROR_RESYNC 2 mirrors 000124 1787138376.864179 write 0x024a41 thru 0x0338f9 (0xeeb9 bytes) 000125 1787138376.913186 MIRROR_RESYNC 2 mirrors 000126 1787138377.416536 MIRROR_RESYNC 2 mirrors 000128 1787138377.697044 read 0x00d9a5 thru 0x011c4b (0x42a7 bytes) 000129 1787138377.704026 trunc from 0x040000 to 0x00d594 (0x32a6b bytes) 000133 1787138377.767735 mapwrite 0x01ed7a thru 0x027da5 (0x902c bytes) 000134 1787138377.832576 mapread 0x0079d3 thru 0x007b96 (0x1c4 bytes) 000138 1787138377.842234 trunc from 0x027da6 to 0x01ca29 (0xb37c bytes) 000144 1787138377.857359 mapread 0x0107ec thru 0x018e94 (0x86a9 bytes) 000145 1787138377.865081 mapwrite 0x03981e thru 0x03ffff (0x67e2 bytes) 000146 1787138377.930411 mapwrite 0x00e0a2 thru 0x01473d (0x669c bytes) 000148 1787138377.986933 read 0x016794 thru 0x019f6a (0x37d7 bytes) 000149 1787138377.987420 trunc from 0x040000 to 0x018078 (0x27f87 bytes) 000150 1787138378.000602 trunc from 0x018079 to 0x026696 (0xe61e bytes) 000151 1787138378.012662 mapwrite 0x01d373 thru 0x01e93f (0x15cd bytes) 000152 1787138378.186757 write 0x03f252 thru 0x03ffff (0xdae bytes) 000154 1787138378.202060 MIRROR_RESYNC 2 mirrors 000156 1787138379.316701 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15c:0x0]: Invalid argument (22) 000157 1787138379.445118 trunc from 0x040000 to 0x03196b (0xe694 bytes) 000160 1787138379.497100 trunc from 0x03196c to 0x0347fd (0x2e92 bytes) 000161 1787138379.512657 trunc from 0x0347fe to 0x01587c (0x1ef81 bytes) 000163 1787138379.532841 read 0x0120ef thru 0x01587c (0x378e bytes) 000164 1787138379.540722 mapread 0x008c82 thru 0x00e0cc (0x544b bytes) 000165 1787138379.559284 mapwrite 0x009db4 thru 0x015a5c (0xbca9 bytes) 000166 1787138379.646404 read 0x015162 thru 0x015a5c (0x8fb bytes) 000167 1787138379.647112 mapwrite 0x00bfb7 thru 0x012e04 (0x6e4e bytes) 000168 1787138379.702948 trunc from 0x015a5d to 0x0226b9 (0xcc5d bytes) 000169 1787138379.723645 write 0x018ff6 thru 0x01ecc9 (0x5cd4 bytes) 000171 1787138379.736503 read 0x00f5fd thru 0x00ffd5 (0x9d9 bytes) 000174 1787138379.739592 mapwrite 0x003648 thru 0x00ccd1 (0x968a bytes) 000180 1787138379.800346 mapwrite 0x02b0fe thru 0x02b6f8 (0x5fb bytes) 000181 1787138379.863685 write 0x0156e3 thru 0x01f2e4 (0x9c02 bytes) 000182 1787138379.867194 trunc from 0x02b6f9 to 0x00c260 (0x1f498 bytes) 000183 1787138379.888052 read 0x00476a thru 0x00580e (0x10a5 bytes) 000185 1787138379.890592 read 0x007f22 thru 0x00c260 (0x433f bytes) 000187 1787138379.892536 trunc from 0x00c261 to 0x0210e0 (0x14e80 bytes) 000188 1787138379.911666 MIRROR_EXTEND to 2 mirrors 000190 1787138380.998005 mapwrite 0x00b368 thru 0x019338 (0xdfd1 bytes) 000191 1787138381.345681 trunc from 0x0210e1 to 0x035788 (0x146a8 bytes) 000192 1787138381.380644 mapwrite 0x03ad29 thru 0x03ffff (0x52d7 bytes) 000193 1787138381.475171 write 0x03a419 thru 0x03ccbc (0x28a4 bytes) 000195 1787138381.478556 mapread 0x02c292 thru 0x03b0a1 (0xee10 bytes) 000196 1787138381.492577 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc6b31c90 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf60aa807 Mirror 3: 0xf60aa807 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000198 1787138382.994614 write 0x00ff3b thru 0x01a752 (0xa818 bytes) 000200 1787138383.058978 mapwrite 0x031d29 thru 0x036f70 (0x5248 bytes) 000202 1787138383.160864 mapread 0x025d0d thru 0x02def1 (0x81e5 bytes) 000203 1787138383.217857 MIRROR_RESYNC 3 mirrors 000204 1787138384.864862 mapwrite 0x02627b thru 0x030704 (0xa48a bytes) 000206 1787138385.147921 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xdd3838ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xdd3838ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000208 1787138385.503027 write 0x01b354 thru 0x0225ec (0x7299 bytes) 000209 1787138385.514085 mapwrite 0x030252 thru 0x039f7d (0x9d2c bytes) 000210 1787138385.610304 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xdd3838ed (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcd86b8a0 Mirror 4: 0xcd86b8a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000211 1787138386.524552 mapwrite 0x0104c6 thru 0x01a471 (0x9fac bytes) 000214 1787138386.706810 mapwrite 0x02183f thru 0x0294a3 (0x7c65 bytes) 000215 1787138386.775658 mapwrite 0x00d99c thru 0x0198a7 (0xbf0c bytes) 000217 1787138386.805577 mapread 0x025817 thru 0x02c06b (0x6855 bytes) 000218 1787138386.820720 MIRROR_RESYNC 3 mirrors 000220 1787138387.745390 read 0x005020 thru 0x014a44 (0xfa25 bytes) 000221 1787138387.764145 trunc from 0x040000 to 0x0247d9 (0x1b826 bytes) 000222 1787138387.866493 trunc from 0x0247da to 0x01634e (0xe48b bytes) 000224 1787138387.900210 mapwrite 0x02de00 thru 0x036211 (0x8412 bytes) 000225 1787138388.016589 MIRROR_RESYNC 3 mirrors 000226 1787138389.413103 MIRROR_EXTEND to 4 mirrors 000228 1787138390.396642 MIRROR_EXTEND to 5 mirrors 000230 1787138391.159072 mapwrite 0x02acda thru 0x02b8fc (0xc23 bytes) 000232 1787138391.490504 mapread 0x0097bb thru 0x01416c (0xa9b2 bytes) 000233 1787138391.545990 mapwrite 0x0263c5 thru 0x026803 (0x43f bytes) 000238 1787138391.626123 read 0x0252c3 thru 0x0321c7 (0xcf05 bytes) 000239 1787138391.627198 mapwrite 0x008bd5 thru 0x010a45 (0x7e71 bytes) 000242 1787138391.716470 read 0x01701b thru 0x026686 (0xf66c bytes) 000243 1787138391.735411 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x36212): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36212): Mirror 3: 0x933ba8f9 (stale) Mirror 4: 0x933ba8f9 (stale) Mirror 5: 0x933ba8f9 (stale) Mirror 6: 0x933ba8f9 (stale) CRC-32 checksum value for chunk [0, 0x36212): Mirror 1: 0xa53a88ba Mirror 7: 0xa53a88ba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36212: skipped 000245 1787138393.721012 trunc from 0x036212 to 0x003b56 (0x326bb bytes) 000247 1787138393.816747 mapread 0x00274b thru 0x003b56 (0x140c bytes) 000249 1787138393.834784 mapread 0x00050d thru 0x003b56 (0x364a bytes) 000250 1787138393.840885 mapread 0x00079c thru 0x003b56 (0x33bb bytes) 000251 1787138393.842083 read 0x0014ef thru 0x001b3c (0x64e bytes) 000252 1787138393.842815 mapread 0x00322c thru 0x003b56 (0x92b bytes) 000253 1787138393.844573 trunc from 0x003b57 to 0x02fd44 (0x2c1ee bytes) 000254 1787138393.875405 write 0x004aec thru 0x013152 (0xe667 bytes) 000257 1787138393.893681 mapwrite 0x02abbd thru 0x032b30 (0x7f74 bytes) 000258 1787138394.024886 mapread 0x018834 thru 0x0259df (0xd1ac bytes) 000259 1787138394.049409 write 0x00638e thru 0x01408c (0xdcff bytes) 000261 1787138394.061809 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x32b31): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32b31): Mirror 3: 0xcb1266bc (stale) Mirror 4: 0xcb1266bc (stale) Mirror 5: 0xcb1266bc (stale) Mirror 6: 0xcb1266bc (stale) Mirror 7: 0xf314f2c5 (stale) CRC-32 checksum value for chunk [0, 0x32b31): Mirror 1: 0xf1596931 Mirror 8: 0xf1596931 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32b31: skipped 000263 1787138395.486289 read 0x00cef7 thru 0x01740e (0xa518 bytes) 000265 1787138395.541309 write 0x022d0e thru 0x02acbf (0x7fb2 bytes) 000268 1787138395.685206 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32b31): Mirror 3: 0xcb1266bc (stale) Mirror 4: 0xcb1266bc (stale) Mirror 5: 0xcb1266bc (stale) Mirror 7: 0xf314f2c5 (stale) Mirror 8: 0xf1596931 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32b31): Mirror 3: 0xcb1266bc (stale) Mirror 4: 0xcb1266bc (stale) Mirror 5: 0xcb1266bc (stale) Mirror 7: 0xf314f2c5 (stale) Mirror 8: 0xf1596931 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32b31: skipped 000269 1787138396.132636 mapread 0x025adc thru 0x02c0aa (0x65cf bytes) 000270 1787138396.163584 mapwrite 0x007f67 thru 0x017eb4 (0xff4e bytes) 000271 1787138396.237049 mapwrite 0x03e940 thru 0x03ffff (0x16c0 bytes) 000272 1787138396.332965 mapwrite 0x020f33 thru 0x02d0eb (0xc1b9 bytes) 000273 1787138396.413903 write 0x003830 thru 0x0118ef (0xe0c0 bytes) 000274 1787138396.416693 write 0x00b7ad thru 0x01503b (0x988f bytes) 000275 1787138396.419492 write 0x01f101 thru 0x02b676 (0xc576 bytes) 000276 1787138396.433615 write 0x01abd1 thru 0x02a2ac (0xf6dc bytes) 000278 1787138396.439208 mapwrite 0x03a8ba thru 0x03ffff (0x5746 bytes) 000279 1787138396.493487 trunc from 0x040000 to 0x02ca0c (0x135f3 bytes) 000280 1787138396.508433 trunc from 0x02ca0d to 0x032df6 (0x63ea bytes) 000281 1787138396.526899 mapwrite 0x02b12a thru 0x03579d (0xa674 bytes) 000283 1787138396.609618 trunc from 0x03579e to 0x014b27 (0x20c76 bytes) 000284 1787138396.636624 write 0x01683c thru 0x01ec4e (0x8413 bytes) 000287 1787138396.643591 trunc from 0x01ec4f to 0x0185cf (0x667f bytes) 000288 1787138396.674628 read 0x00823d thru 0x014cca (0xca8e bytes) 000290 1787138396.682212 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x185d0): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x185d0): Mirror 3: 0x530a0d2d (stale) Mirror 4: 0x530a0d2d (stale) Mirror 5: 0x530a0d2d (stale) Mirror 7: 0xbfd41534 (stale) Mirror 8: 0x685b6b7a (stale) CRC-32 checksum value for chunk [0, 0x185d0): Mirror 1: 0x46a46ce4 Mirror 9: 0x46a46ce4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x185d0: skipped 000291 1787138397.769603 read 0x015753 thru 0x0185cf (0x2e7d bytes) 000292 1787138397.790318 mapwrite 0x01ac7f thru 0x0281a8 (0xd52a bytes) 000293 1787138398.088951 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x281a9): Mirror 3: 0x29deb127 (stale) Mirror 5: 0x29deb127 (stale) Mirror 7: 0x1caf7a44 (stale) Mirror 8: 0x8e65ab5a (stale) Mirror 9: 0x3347ee80 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x281a9): Mirror 3: 0x29deb127 (stale) Mirror 5: 0x29deb127 (stale) Mirror 7: 0x1caf7a44 (stale) Mirror 8: 0x8e65ab5a (stale) Mirror 9: 0x3347ee80 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x281a9: skipped 000294 1787138398.538736 mapwrite 0x016d21 thru 0x020df7 (0xa0d7 bytes) 000296 1787138398.622144 trunc from 0x0281a9 to 0x02eb71 (0x69c9 bytes) 000299 1787138398.650769 write 0x01532a thru 0x01c80a (0x74e1 bytes) 000302 1787138398.654313 mapread 0x026e52 thru 0x02aa30 (0x3bdf bytes) 000303 1787138398.669654 read 0x024f7b thru 0x026a15 (0x1a9b bytes) 000305 1787138398.670492 mapwrite 0x026abe thru 0x0351fb (0xe73e bytes) 000308 1787138398.836426 mapread 0x0319c7 thru 0x0351fb (0x3835 bytes) 000309 1787138398.837977 mapread 0x01a4fc thru 0x028c96 (0xe79b bytes) 000311 1787138398.849421 mapread 0x024d23 thru 0x02faf1 (0xadcf bytes) 000312 1787138398.851827 trunc from 0x0351fc to 0x00a2a1 (0x2af5a bytes) 000314 1787138398.898166 mapwrite 0x038263 thru 0x03ffff (0x7d9d bytes) 000315 1787138399.029347 write 0x008160 thru 0x008ff4 (0xe95 bytes) 000316 1787138399.039170 read 0x023be4 thru 0x02a8f7 (0x6d14 bytes) 000317 1787138399.077491 MIRROR_RESYNC 6 mirrors 000321 1787138401.155197 read 0x03aee5 thru 0x03ffff (0x511b bytes) 000322 1787138401.165938 trunc from 0x040000 to 0x0317ae (0xe851 bytes) 000324 1787138401.220527 read 0x013b3e thru 0x0209e3 (0xcea6 bytes) 000325 1787138401.231506 write 0x03531f thru 0x03a7bb (0x549d bytes) 000326 1787138401.238596 mapread 0x034088 thru 0x03a7bb (0x6734 bytes) 000329 1787138401.255861 trunc from 0x03a7bc to 0x00ff6b (0x2a850 bytes) 000330 1787138401.289052 read 0x006f7a thru 0x00f212 (0x8299 bytes) 000331 1787138401.291963 mapwrite 0x012406 thru 0x01f1c2 (0xcdbd bytes) 000332 1787138401.495717 read 0x019c99 thru 0x01f1c2 (0x552a bytes) 000334 1787138401.510548 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f1c3): Mirror 5: 0xdd7f2712 (stale) Mirror 7: 0xdd7f2712 (stale) Mirror 8: 0xdd7f2712 (stale) Mirror 9: 0xdd7f2712 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f1c3): Mirror 5: 0xdd7f2712 (stale) Mirror 7: 0xdd7f2712 (stale) Mirror 8: 0xdd7f2712 (stale) Mirror 9: 0xdd7f2712 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f1c3: skipped 000335 1787138402.021415 mapwrite 0x00038d thru 0x00e608 (0xe27c bytes) 000336 1787138402.120181 read 0x004487 thru 0x00fc48 (0xb7c2 bytes) 000337 1787138402.122081 write 0x02d407 thru 0x031da4 (0x499e bytes) 000339 1787138402.125767 read 0x006756 thru 0x00a1b0 (0x3a5b bytes) 000340 1787138402.126829 mapread 0x01730f thru 0x0241df (0xced1 bytes) 000341 1787138402.151762 MIRROR_RESYNC 5 mirrors 000342 1787138403.897142 read 0x017573 thru 0x0198fd (0x238b bytes) 000343 1787138403.903180 read 0x021a72 thru 0x02a787 (0x8d16 bytes) 000344 1787138403.915670 mapread 0x031850 thru 0x031da4 (0x555 bytes) truncating to largest ever: 0x3f864 000345 1787138403.917686 trunc from 0x031da5 to 0x03f863 (0xdabf bytes) 000346 1787138403.994051 read 0x03f10c thru 0x03f863 (0x758 bytes) 000348 1787138404.010151 mapwrite 0x03d2a7 thru 0x03ffff (0x2d59 bytes) 000349 1787138404.150285 write 0x0153a1 thru 0x0191e2 (0x3e42 bytes) 000350 1787138404.155017 mapread 0x006ae0 thru 0x00ec53 (0x8174 bytes) 000352 1787138404.161682 write 0x00faf5 thru 0x01002a (0x536 bytes) 000354 1787138404.169658 MIRROR_RESYNC 5 mirrors 000355 1787138405.676845 read 0x00361f thru 0x013451 (0xfe33 bytes) 000358 1787138405.688126 mapwrite 0x025368 thru 0x031ac5 (0xc75e bytes) 000360 1787138405.910764 trunc from 0x040000 to 0x014102 (0x2befd bytes) 000363 1787138405.937786 mapread 0x002301 thru 0x0061e4 (0x3ee4 bytes) 000365 1787138405.971763 mapread 0x003a22 thru 0x004ff7 (0x15d6 bytes) 000366 1787138405.980784 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14103): Mirror 5: 0xceba3280 (stale) Mirror 7: 0xceba3280 (stale) Mirror 8: 0xceba3280 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14103): Mirror 5: 0xceba3280 (stale) Mirror 7: 0xceba3280 (stale) Mirror 8: 0xceba3280 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14103: skipped 000367 1787138406.342860 mapwrite 0x002686 thru 0x00e02c (0xb9a7 bytes) 000368 1787138406.430644 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14103): Mirror 5: 0xceba3280 (stale) Mirror 7: 0xceba3280 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14103): Mirror 5: 0xceba3280 (stale) Mirror 7: 0xceba3280 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14103: skipped 000370 1787138406.779926 mapread 0x004e3b thru 0x00bc47 (0x6e0d bytes) 000371 1787138406.812514 MIRROR_RESYNC 3 mirrors 000372 1787138407.734335 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15c:0x0]: Invalid argument (22) 000375 1787138408.052365 read 0x00429a thru 0x011daa (0xdb11 bytes) 000376 1787138408.071014 mapread 0x008d53 thru 0x00f1ca (0x6478 bytes) 000377 1787138408.072033 read 0x004a98 thru 0x011325 (0xc88e bytes) 000378 1787138408.077867 MIRROR_RESYNC 2 mirrors 000382 1787138408.482467 trunc from 0x014103 to 0x00ed5f (0x53a3 bytes) 000384 1787138408.572438 trunc from 0x00ed60 to 0x00ebd0 (0x18f bytes) 000385 1787138408.598786 trunc from 0x00ebd1 to 0x036887 (0x27cb7 bytes) 000388 1787138408.615471 read 0x00aeaf thru 0x0144a2 (0x95f4 bytes) 000393 1787138408.633045 mapread 0x01a0a2 thru 0x022d11 (0x8c70 bytes) 000394 1787138408.653570 mapwrite 0x027c42 thru 0x02e3dd (0x679c bytes) 000395 1787138408.766377 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15c:0x0]: Invalid argument (22) 000396 1787138408.960695 write 0x02cb1e thru 0x0354ef (0x89d2 bytes) 000397 1787138408.975675 write 0x02fdfd thru 0x0386e4 (0x88e8 bytes) 000402 1787138408.981986 mapwrite 0x03e1a7 thru 0x03ffff (0x1e59 bytes) 000403 1787138409.073428 trunc from 0x040000 to 0x02718d (0x18e72 bytes) 000404 1787138409.085390 trunc from 0x02718e to 0x034b35 (0xd9a8 bytes) 000405 1787138409.096672 mapread 0x014a12 thru 0x015681 (0xc70 bytes) 000406 1787138409.107727 trunc from 0x034b36 to 0x025a8c (0xf0a9 bytes) 000407 1787138409.121089 write 0x02aebf thru 0x0343ab (0x94ed bytes) 000413 1787138409.127029 read 0x014a35 thru 0x01513b (0x707 bytes) 000414 1787138409.130856 mapwrite 0x02c054 thru 0x0385fd (0xc5aa bytes) 000419 1787138409.226443 read 0x020210 thru 0x022c5a (0x2a4b bytes) 000421 1787138409.245841 write 0x03e26e thru 0x03ffff (0x1d92 bytes) 000422 1787138409.246954 mapread 0x03509a thru 0x0387e4 (0x374b bytes) 000425 1787138409.249888 mapread 0x030f70 thru 0x03a543 (0x95d4 bytes) 000428 1787138409.267972 trunc from 0x040000 to 0x02f3fd (0x10c02 bytes) 000430 1787138409.302021 trunc from 0x02f3fe to 0x033643 (0x4246 bytes) 000432 1787138409.320051 read 0x002c97 thru 0x006d58 (0x40c2 bytes) 000433 1787138409.321193 write 0x02171a thru 0x0226cc (0xfb3 bytes) 000434 1787138409.322707 read 0x002183 thru 0x002b8f (0xa0d bytes) 000435 1787138409.323303 mapread 0x022dec thru 0x028107 (0x531c bytes) 000438 1787138409.325575 write 0x0019fe thru 0x00609d (0x46a0 bytes) 000439 1787138409.326639 read 0x013955 thru 0x019510 (0x5bbc bytes) 000440 1787138409.327548 mapread 0x022358 thru 0x029373 (0x701c bytes) 000441 1787138409.328909 mapread 0x02c466 thru 0x0319af (0x554a bytes) 000444 1787138409.348664 MIRROR_EXTEND to 2 mirrors 000446 1787138410.401667 read 0x0139b9 thru 0x023826 (0xfe6e bytes) 000447 1787138410.423365 trunc from 0x033644 to 0x03cfb0 (0x996d bytes) 000448 1787138410.625277 write 0x017741 thru 0x01932f (0x1bef bytes) 000449 1787138410.657072 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15c:0x0]: Invalid argument (22) 000451 1787138410.979411 mapwrite 0x02d255 thru 0x03c817 (0xf5c3 bytes) 000458 1787138411.165689 write 0x029267 thru 0x033962 (0xa6fc bytes) 000460 1787138411.177018 trunc from 0x03cfb1 to 0x010fc9 (0x2bfe7 bytes) 000461 1787138411.222270 write 0x034285 thru 0x03ffff (0xbd7b bytes) 000462 1787138411.223901 write 0x011fef thru 0x01bd30 (0x9d42 bytes) 000463 1787138411.239064 mapwrite 0x00d86c thru 0x00fd84 (0x2519 bytes) 000464 1787138411.341526 mapwrite 0x03e8d7 thru 0x03ffff (0x1729 bytes) 000465 1787138411.391450 read 0x02ca0d thru 0x034f17 (0x850b bytes) 000466 1787138411.405475 mapwrite 0x010029 thru 0x01839e (0x8376 bytes) 000467 1787138411.466708 trunc from 0x040000 to 0x0150de (0x2af21 bytes) 000468 1787138411.489024 write 0x033360 thru 0x038434 (0x50d5 bytes) 000471 1787138411.493507 mapread 0x011b3f thru 0x016719 (0x4bdb bytes) 000472 1787138411.506575 mapread 0x02e7ee thru 0x031352 (0x2b65 bytes) 000473 1787138411.521655 mapwrite 0x02e1c1 thru 0x02ece4 (0xb24 bytes) 000474 1787138411.618698 mapwrite 0x032295 thru 0x03ffff (0xdd6b bytes) 000475 1787138411.703027 mapwrite 0x032e57 thru 0x03e019 (0xb1c3 bytes) 000479 1787138411.756391 mapread 0x003e77 thru 0x0040b4 (0x23e bytes) 000481 1787138411.768119 read 0x01e8b8 thru 0x0290f1 (0xa83a bytes) 000482 1787138411.780049 write 0x0300e2 thru 0x03f4f6 (0xf415 bytes) 000484 1787138411.783122 read 0x0092fe thru 0x0177f0 (0xe4f3 bytes) 000486 1787138411.792857 MIRROR_EXTEND to 2 mirrors 000487 1787138412.823658 mapwrite 0x03b72d thru 0x03ffff (0x48d3 bytes) 000489 1787138413.044013 write 0x027565 thru 0x030b51 (0x95ed bytes) 000490 1787138413.065047 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf5ee268c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x39b0ec92 Mirror 3: 0x39b0ec92 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000492 1787138414.590623 mapwrite 0x01fbe9 thru 0x028b28 (0x8f40 bytes) 000494 1787138414.878322 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x39b0ec92 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x39b0ec92 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000496 1787138415.454604 mapwrite 0x01a15a thru 0x021d9d (0x7c44 bytes) 000497 1787138415.629309 write 0x02f2e2 thru 0x03a803 (0xb522 bytes) 000498 1787138415.656136 read 0x037d14 thru 0x03d5ef (0x58dc bytes) 000500 1787138415.668948 trunc from 0x040000 to 0x03e9bf (0x1640 bytes) 000501 1787138415.709662 mapread 0x007729 thru 0x00c4b0 (0x4d88 bytes) 000502 1787138415.719767 trunc from 0x03e9c0 to 0x0275bd (0x17402 bytes) 000503 1787138415.765238 write 0x026881 thru 0x031419 (0xab99 bytes) 000505 1787138415.781825 mapread 0x00f491 thru 0x0198b1 (0xa421 bytes) 000506 1787138415.797736 mapread 0x01a546 thru 0x0273d4 (0xce8f bytes) 000507 1787138415.811013 trunc from 0x03141a to 0x02d286 (0x4193 bytes) 000509 1787138415.846623 mapread 0x00fae8 thru 0x0101d9 (0x6f2 bytes) 000510 1787138415.847921 mapwrite 0x011281 thru 0x0139f8 (0x2778 bytes) 000512 1787138415.994031 trunc from 0x02d287 to 0x031a62 (0x47dc bytes) 000515 1787138416.031138 mapread 0x029034 thru 0x0317a1 (0x876e bytes) 000516 1787138416.045626 mapwrite 0x0174de thru 0x0196ed (0x2210 bytes) 000517 1787138416.111014 MIRROR_RESYNC 2 mirrors 000518 1787138417.045171 mapwrite 0x007a48 thru 0x01709c (0xf655 bytes) 000519 1787138417.307643 mapwrite 0x01c9bf thru 0x024ffc (0x863e bytes) 000520 1787138417.423626 read 0x02b70b thru 0x02e4af (0x2da5 bytes) 000521 1787138417.440486 MIRROR_RESYNC 2 mirrors 000522 1787138418.275755 trunc from 0x031a63 to 0x03ecac (0xd24a bytes) 000523 1787138418.428891 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ecad): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ecad): Mirror 3: 0xd6505fe6 (stale) CRC-32 checksum value for chunk [0, 0x3ecad): Mirror 1: 0xd6505fe6 Mirror 4: 0xd6505fe6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ecad: skipped 000525 1787138419.897599 mapwrite 0x00f225 thru 0x01d687 (0xe463 bytes) 000526 1787138420.193717 mapread 0x027d22 thru 0x0291e5 (0x14c4 bytes) 000527 1787138420.205367 write 0x01138c thru 0x011b93 (0x808 bytes) 000528 1787138420.208319 read 0x003b25 thru 0x008761 (0x4c3d bytes) 000529 1787138420.221043 trunc from 0x03ecad to 0x02da8e (0x1121e bytes) 000531 1787138420.256983 MIRROR_RESYNC 3 mirrors 000533 1787138421.130476 mapwrite 0x035b29 thru 0x036d19 (0x11f1 bytes) 000534 1787138421.352857 trunc from 0x036d1a to 0x0014b9 (0x35860 bytes) 000537 1787138421.393211 write 0x00f387 thru 0x014578 (0x51f2 bytes) 000545 1787138421.400552 write 0x01b1b8 thru 0x02113c (0x5f85 bytes) 000547 1787138421.403690 trunc from 0x02113d to 0x0129b2 (0xe78a bytes) 000548 1787138421.432134 MIRROR_RESYNC 3 mirrors 000549 1787138422.504636 mapread 0x0070a9 thru 0x00c73a (0x5692 bytes) 000550 1787138422.533275 trunc from 0x0129b3 to 0x0149a2 (0x1ff0 bytes) 000554 1787138422.638905 mapwrite 0x00aa5f thru 0x011763 (0x6d05 bytes) 000555 1787138422.717866 trunc from 0x0149a3 to 0x016f31 (0x258f bytes) 000556 1787138422.752513 write 0x035c6e thru 0x03ba63 (0x5df6 bytes) 000557 1787138422.760222 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ba64): Mirror 4: 0x606d05a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba64): Mirror 4: 0x606d05a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba64: skipped 000559 1787138423.071621 mapwrite 0x00074e thru 0x009279 (0x8b2c bytes) 000562 1787138423.184647 read 0x02340b thru 0x02344f (0x45 bytes) 000563 1787138423.190459 mapwrite 0x028ec4 thru 0x030e58 (0x7f95 bytes) 000564 1787138423.264622 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ba64): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba64): Mirror 4: 0x606d05a (stale) CRC-32 checksum value for chunk [0, 0x3ba64): Mirror 1: 0x59951f7d Mirror 5: 0x59951f7d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba64: skipped 000565 1787138424.504852 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15c:0x0]: Invalid argument (22) 000566 1787138425.060179 MIRROR_RESYNC 2 mirrors 000568 1787138425.656962 read 0x039b9a thru 0x03ba63 (0x1eca bytes) 000570 1787138425.710541 mapread 0x00bec5 thru 0x019f2b (0xe067 bytes) 000571 1787138425.789718 mapwrite 0x012266 thru 0x0202eb (0xe086 bytes) 000572 1787138425.995917 write 0x0032f3 thru 0x00886c (0x557a bytes) 000573 1787138426.005664 write 0x01da1b thru 0x01f5b3 (0x1b99 bytes) 000574 1787138426.007692 write 0x0148f2 thru 0x018c77 (0x4386 bytes) 000576 1787138426.012507 trunc from 0x03ba64 to 0x0387fb (0x3268 bytes) 000581 1787138426.057444 write 0x02722b thru 0x02ea5a (0x7830 bytes) 000584 1787138426.072300 MIRROR_RESYNC 2 mirrors 000585 1787138427.113798 read 0x02015e thru 0x02496c (0x480f bytes) 000586 1787138427.136684 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15c:0x0]: Invalid argument (22) 000587 1787138427.451493 write 0x012d6a thru 0x01531e (0x25b5 bytes) 000588 1787138427.489038 read 0x00d5e6 thru 0x018f60 (0xb97b bytes) 000589 1787138427.513746 mapwrite 0x024038 thru 0x02e868 (0xa831 bytes) 000593 1787138427.799538 write 0x03edd0 thru 0x03f548 (0x779 bytes) 000595 1787138427.811870 read 0x0182c7 thru 0x01f9d1 (0x770b bytes) 000597 1787138427.854864 mapwrite 0x02003b thru 0x02e480 (0xe446 bytes) 000600 1787138427.953356 mapwrite 0x01ec28 thru 0x02d6fb (0xead4 bytes) 000603 1787138428.023385 write 0x0296fa thru 0x031103 (0x7a0a bytes) 000605 1787138428.028627 write 0x00bbd9 thru 0x018931 (0xcd59 bytes) 000606 1787138428.032932 trunc from 0x03f549 to 0x027967 (0x17be1 bytes) 000607 1787138428.069916 write 0x006b8d thru 0x00e66a (0x7ade bytes) 000608 1787138428.073607 mapread 0x01cda2 thru 0x027967 (0xabc6 bytes) 000610 1787138428.078142 mapwrite 0x02bf6a thru 0x039f0a (0xdfa1 bytes) 000611 1787138428.396642 read 0x00cdb1 thru 0x01b92b (0xeb7b bytes) 000614 1787138428.401510 trunc from 0x039f0b to 0x0248fa (0x15610 bytes) 000615 1787138428.420548 trunc from 0x0248fb to 0x019ee1 (0xaa19 bytes) 000616 1787138428.438342 trunc from 0x019ee2 to 0x001e5a (0x18087 bytes) 000617 1787138428.458794 mapwrite 0x0352e3 thru 0x03ffff (0xad1d bytes) 000620 1787138428.563403 mapread 0x02d578 thru 0x03ae3c (0xd8c5 bytes) 000623 1787138428.566172 write 0x03588f thru 0x039646 (0x3db8 bytes) 000626 1787138428.569604 trunc from 0x040000 to 0x004bfb (0x3b404 bytes) 000627 1787138428.593891 mapwrite 0x022f09 thru 0x02a36e (0x7466 bytes) 000628 1787138428.753659 write 0x03a046 thru 0x03ffff (0x5fba bytes) 000629 1787138428.756369 trunc from 0x040000 to 0x01993f (0x266c0 bytes) 000631 1787138428.779625 mapwrite 0x03adf0 thru 0x03ffff (0x5210 bytes) 000632 1787138428.893327 trunc from 0x040000 to 0x03961e (0x69e1 bytes) 000633 1787138428.909499 trunc from 0x03961f to 0x02d9d8 (0xbc46 bytes) 000634 1787138428.947166 mapwrite 0x0367f7 thru 0x0379d6 (0x11e0 bytes) 000638 1787138429.021134 mapwrite 0x03c507 thru 0x03ffff (0x3af9 bytes) 000640 1787138429.171813 write 0x01f3c8 thru 0x022dfd (0x3a36 bytes) 000641 1787138429.189533 read 0x004b19 thru 0x00fd3a (0xb222 bytes) 000642 1787138429.206978 read 0x0112f2 thru 0x012a62 (0x1771 bytes) 000647 1787138429.216507 trunc from 0x040000 to 0x016f40 (0x290bf bytes) 000648 1787138429.239725 mapwrite 0x021e47 thru 0x02945a (0x7614 bytes) 000650 1787138429.382493 mapread 0x00812d thru 0x00a711 (0x25e5 bytes) 000652 1787138429.387107 read 0x028d29 thru 0x02945a (0x732 bytes) 000654 1787138429.391544 write 0x007c43 thru 0x015540 (0xd8fe bytes) 000657 1787138429.395641 mapwrite 0x037cc1 thru 0x03ffff (0x833f bytes) 000660 1787138429.494127 trunc from 0x040000 to 0x037ad2 (0x852d bytes) 000662 1787138429.521635 MIRROR_EXTEND to 2 mirrors 000663 1787138430.366696 read 0x03059b thru 0x037ad2 (0x7538 bytes) 000664 1787138430.382506 read 0x01cb33 thru 0x026cb0 (0xa17e bytes) 000665 1787138430.410696 trunc from 0x037ad3 to 0x033d15 (0x3dbd bytes) 000667 1787138430.501421 mapread 0x004998 thru 0x0064d5 (0x1b3e bytes) 000671 1787138430.519661 write 0x01ea51 thru 0x026045 (0x75f5 bytes) 000676 1787138430.534153 mapread 0x00e004 thru 0x015f3e (0x7f3b bytes) 000677 1787138430.583599 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15c:0x0]: Invalid argument (22) 000679 1787138430.801838 mapwrite 0x020350 thru 0x02f6eb (0xf39c bytes) 000680 1787138430.936556 trunc from 0x033d16 to 0x00966d (0x2a6a8 bytes) 000681 1787138430.970867 mapread 0x003b14 thru 0x00790a (0x3df7 bytes) 000683 1787138430.991066 read 0x007302 thru 0x00966d (0x236c bytes) 000684 1787138430.993398 write 0x0262b5 thru 0x030371 (0xa0bd bytes) 000686 1787138431.007333 mapread 0x00e6e6 thru 0x01647d (0x7d98 bytes) 000687 1787138431.018672 mapwrite 0x037c8d thru 0x03ffff (0x8373 bytes) 000688 1787138431.166851 read 0x01cd74 thru 0x02070f (0x399c bytes) 000689 1787138431.175154 read 0x02f8d9 thru 0x03e62e (0xed56 bytes) 000691 1787138431.176808 trunc from 0x040000 to 0x016b60 (0x2949f bytes) 000692 1787138431.196192 write 0x01df93 thru 0x01efbb (0x1029 bytes) 000693 1787138431.198741 mapread 0x0042de thru 0x008afd (0x4820 bytes) 000694 1787138431.202135 read 0x00b89b thru 0x00df5f (0x26c5 bytes) 000695 1787138431.202897 read 0x0086a0 thru 0x01088a (0x81eb bytes) 000696 1787138431.207994 MIRROR_EXTEND to 2 mirrors 000699 1787138432.397517 write 0x01bb85 thru 0x029bbc (0xe038 bytes) 000700 1787138432.485286 trunc from 0x029bbd to 0x02c976 (0x2dba bytes) 000701 1787138432.527700 trunc from 0x02c977 to 0x01893a (0x1403c bytes) 000702 1787138432.548600 read 0x0030c5 thru 0x004978 (0x18b4 bytes) 000704 1787138432.585687 mapread 0x004276 thru 0x00d02e (0x8db9 bytes) 000705 1787138432.640794 mapread 0x0166f3 thru 0x01893a (0x2248 bytes) 000706 1787138432.663453 mapread 0x005d02 thru 0x0120dc (0xc3db bytes) 000707 1787138432.666715 mapread 0x0121d3 thru 0x01893a (0x6768 bytes) 000708 1787138432.670593 mapread 0x0075fb thru 0x011956 (0xa35c bytes) 000709 1787138432.675617 write 0x00ae2c thru 0x01266f (0x7844 bytes) 000711 1787138432.687796 mapwrite 0x01ba22 thru 0x029752 (0xdd31 bytes) 000714 1787138432.778152 trunc from 0x029753 to 0x01412e (0x15624 bytes) 000716 1787138432.822500 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15c:0x0]: Invalid argument (22) 000717 1787138433.039820 read 0x00943f thru 0x01412e (0xacf0 bytes) 000718 1787138433.055287 write 0x00ca95 thru 0x010e68 (0x43d4 bytes) 000721 1787138433.060047 mapread 0x001e2c thru 0x0094cd (0x76a2 bytes) 000723 1787138433.085823 trunc from 0x01412f to 0x03b2e7 (0x271b9 bytes) 000724 1787138433.110773 mapread 0x019ab6 thru 0x021f75 (0x84c0 bytes) 000725 1787138433.133643 MIRROR_EXTEND to 2 mirrors 000727 1787138434.464529 mapread 0x039066 thru 0x03b2e7 (0x2282 bytes) 000728 1787138434.580178 read 0x004006 thru 0x00e4c5 (0xa4c0 bytes) 000729 1787138434.628577 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15c:0x0]: Invalid argument (22) 000730 1787138434.896057 read 0x00439b thru 0x012399 (0xdfff bytes) 000732 1787138434.908037 mapwrite 0x0264ed thru 0x029c33 (0x3747 bytes) 000733 1787138434.994012 mapwrite 0x033d92 thru 0x035d1e (0x1f8d bytes) 000736 1787138435.080233 mapwrite 0x023f9b thru 0x033f5e (0xffc4 bytes) 000738 1787138435.126512 MIRROR_EXTEND to 2 mirrors 000739 1787138436.310640 trunc from 0x03b2e8 to 0x015389 (0x25f5e bytes) 000740 1787138436.398353 mapwrite 0x03c543 thru 0x03ffff (0x3abd bytes) 000741 1787138436.612664 trunc from 0x040000 to 0x0328a6 (0xd759 bytes) 000744 1787138436.661073 trunc from 0x0328a7 to 0x007d30 (0x2ab76 bytes) 000746 1787138436.706041 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15c:0x0]: Invalid argument (22) 000747 1787138436.859168 trunc from 0x007d31 to 0x023670 (0x1b940 bytes) 000749 1787138436.874634 read 0x01a71c thru 0x023670 (0x8f55 bytes) 000750 1787138436.885453 write 0x02dffc thru 0x036eb0 (0x8eb5 bytes) 000752 1787138436.888850 read 0x012163 thru 0x021fac (0xfe4a bytes) 000753 1787138436.907482 write 0x011df2 thru 0x012a84 (0xc93 bytes) 000754 1787138436.914539 write 0x02b0f5 thru 0x032c70 (0x7b7c bytes) 000755 1787138436.916190 mapread 0x02c655 thru 0x02f40d (0x2db9 bytes) 000756 1787138436.924569 mapread 0x02244a thru 0x026f9d (0x4b54 bytes) 000760 1787138436.935631 trunc from 0x036eb1 to 0x014935 (0x2257b bytes) 000764 1787138436.981417 mapwrite 0x00cd5e thru 0x01585b (0x8afe bytes) 000768 1787138437.109511 read 0x006713 thru 0x00c124 (0x5a12 bytes) 000769 1787138437.110604 mapread 0x00740b thru 0x00b7fd (0x43f3 bytes) 000770 1787138437.114523 mapwrite 0x032fd3 thru 0x0389f4 (0x5a22 bytes) 000772 1787138437.214428 mapread 0x0373c1 thru 0x0389f4 (0x1634 bytes) 000774 1787138437.218237 mapread 0x017b1c thru 0x02055f (0x8a44 bytes) 000775 1787138437.225071 MIRROR_EXTEND to 2 mirrors 000776 1787138438.073484 read 0x0373ae thru 0x0389f4 (0x1647 bytes) 000777 1787138438.091926 read 0x0074be thru 0x00ddf5 (0x6938 bytes) 000778 1787138438.116211 write 0x01ebec thru 0x02b15a (0xc56f bytes) 000781 1787138438.177462 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15c:0x0]: Invalid argument (22) 000784 1787138438.353787 MIRROR_EXTEND to 2 mirrors 000785 1787138439.143804 mapread 0x02177d thru 0x02f162 (0xd9e6 bytes) 000787 1787138439.202705 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15c:0x0]: Invalid argument (22) 000789 1787138439.342687 write 0x03889e thru 0x03ffff (0x7762 bytes) 000790 1787138439.364531 read 0x01fd25 thru 0x02c6cb (0xc9a7 bytes) 000793 1787138439.377493 write 0x02c682 thru 0x02d334 (0xcb3 bytes) 000796 1787138439.384176 mapwrite 0x002f35 thru 0x006134 (0x3200 bytes) 000798 1787138439.560193 MIRROR_EXTEND to 2 mirrors 000799 1787138440.232185 read 0x00abca thru 0x01a6cb (0xfb02 bytes) 000801 1787138440.258148 read 0x00fbab thru 0x01003c (0x492 bytes) 000803 1787138440.278936 trunc from 0x040000 to 0x030125 (0xfeda bytes) 000805 1787138440.479515 trunc from 0x030126 to 0x036445 (0x6320 bytes) 000806 1787138440.504651 mapwrite 0x02b050 thru 0x02f97a (0x492b bytes) 000810 1787138440.561601 write 0x0349fe thru 0x03ffff (0xb602 bytes) 000811 1787138440.568407 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15c:0x0]: Invalid argument (22) 000812 1787138440.718689 mapread 0x02a708 thru 0x02b264 (0xb5d bytes) 000813 1787138440.731645 write 0x024b38 thru 0x029dd9 (0x52a2 bytes) 000814 1787138440.743559 mapwrite 0x02afa1 thru 0x030ecc (0x5f2c bytes) 000815 1787138440.812089 mapread 0x03ea9e thru 0x03ffff (0x1562 bytes) 000817 1787138440.825631 trunc from 0x040000 to 0x02fdab (0x10254 bytes) 000821 1787138440.843902 mapwrite 0x010b3d thru 0x017140 (0x6604 bytes) 000822 1787138440.929305 trunc from 0x02fdac to 0x026b70 (0x923b bytes) 000823 1787138440.954684 mapread 0x022d03 thru 0x026b70 (0x3e6e bytes) 000824 1787138440.966602 write 0x0182ef thru 0x01dfac (0x5cbe bytes) 000825 1787138440.967887 mapwrite 0x03fa00 thru 0x03ffff (0x600 bytes) 000829 1787138441.047430 trunc from 0x040000 to 0x0020df (0x3df20 bytes) 000830 1787138441.072835 mapwrite 0x01e406 thru 0x025b51 (0x774c bytes) 000834 1787138441.204734 write 0x02411d thru 0x033ac3 (0xf9a7 bytes) 000837 1787138441.212475 mapwrite 0x0331c9 thru 0x03c488 (0x92c0 bytes) 000838 1787138441.308073 read 0x01cba4 thru 0x02214c (0x55a9 bytes) 000842 1787138441.314208 trunc from 0x03c489 to 0x022017 (0x1a471 bytes) 000843 1787138441.332796 read 0x01fc15 thru 0x0206f2 (0xade bytes) 000844 1787138441.333347 mapwrite 0x02f561 thru 0x03b207 (0xbca7 bytes) 000845 1787138441.454356 write 0x006541 thru 0x011fde (0xba9e bytes) 000846 1787138441.455804 read 0x0264b3 thru 0x028b9b (0x26e9 bytes) 000855 1787138441.465479 mapwrite 0x018ee2 thru 0x01db85 (0x4ca4 bytes) 000856 1787138441.498087 trunc from 0x03b208 to 0x00ac43 (0x305c4 bytes) 000857 1787138441.510687 read 0x0052e7 thru 0x00ac43 (0x595d bytes) 000858 1787138441.511597 mapread 0x00a0d2 thru 0x00ac43 (0xb72 bytes) 000860 1787138441.513304 read 0x000e48 thru 0x00ac43 (0x9dfc bytes) 000865 1787138441.515435 mapread 0x00101e thru 0x005724 (0x4707 bytes) 000866 1787138441.516152 trunc from 0x00ac44 to 0x031313 (0x266d0 bytes) 000869 1787138441.530212 read 0x00d15d thru 0x016b43 (0x99e7 bytes) 000870 1787138441.537569 mapread 0x024970 thru 0x02a5f3 (0x5c84 bytes) 000871 1787138441.538466 read 0x028951 thru 0x031313 (0x89c3 bytes) 000872 1787138441.539128 trunc from 0x031314 to 0x035e72 (0x4b5f bytes) 000873 1787138441.550060 write 0x022f3e thru 0x02cfe5 (0xa0a8 bytes) 000874 1787138441.551103 mapwrite 0x004bcb thru 0x013394 (0xe7ca bytes) 000875 1787138441.597051 mapwrite 0x004afe thru 0x00bb2b (0x702e bytes) 000876 1787138441.649148 read 0x02259f thru 0x024fb9 (0x2a1b bytes) 000878 1787138441.656103 MIRROR_EXTEND to 2 mirrors 000881 1787138442.709149 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15c:0x0]: Invalid argument (22) 000884 1787138442.883321 mapread 0x025139 thru 0x02ae27 (0x5cef bytes) 000885 1787138442.916998 write 0x004157 thru 0x006d07 (0x2bb1 bytes) 000886 1787138442.960554 mapwrite 0x0330a0 thru 0x03ac7f (0x7be0 bytes) 000888 1787138443.098122 mapread 0x00d428 thru 0x019587 (0xc160 bytes) 000890 1787138443.119235 write 0x02271f thru 0x02d9be (0xb2a0 bytes) 000891 1787138443.135789 write 0x025b3d thru 0x027eee (0x23b2 bytes) 000892 1787138443.139571 mapwrite 0x014fc9 thru 0x01def8 (0x8f30 bytes) 000893 1787138443.226542 mapwrite 0x021ea1 thru 0x03096d (0xeacd bytes) 000894 1787138443.292010 trunc from 0x03ac80 to 0x029f85 (0x10cfa bytes) 000898 1787138443.332132 write 0x0176b4 thru 0x026534 (0xee81 bytes) 000900 1787138443.336080 write 0x019b69 thru 0x01e9c5 (0x4e5d bytes) 000901 1787138443.338404 mapwrite 0x03ee57 thru 0x03ffff (0x11a9 bytes) 000902 1787138443.496490 trunc from 0x040000 to 0x009431 (0x36bce bytes) 000903 1787138443.541881 write 0x02d555 thru 0x037608 (0xa0b4 bytes) 000904 1787138443.544058 trunc from 0x037609 to 0x0135e5 (0x24023 bytes) 000906 1787138443.585352 mapwrite 0x028ea1 thru 0x037791 (0xe8f1 bytes) 000909 1787138443.771263 write 0x009bb0 thru 0x010754 (0x6ba5 bytes) 000910 1787138443.783134 trunc from 0x037792 to 0x02aa97 (0xccfa bytes) 000913 1787138443.821652 mapwrite 0x030104 thru 0x03120f (0x110c bytes) 000916 1787138443.993132 mapread 0x01c603 thru 0x01da7f (0x147d bytes) 000917 1787138444.004584 mapread 0x01e309 thru 0x0212ef (0x2fe7 bytes) 000918 1787138444.005625 write 0x022b95 thru 0x02e15d (0xb5c9 bytes) 000923 1787138444.012055 mapread 0x011c2f thru 0x020c06 (0xefd8 bytes) 000925 1787138444.022045 read 0x00f539 thru 0x017641 (0x8109 bytes) 000926 1787138444.022881 read 0x01d93b thru 0x01e981 (0x1047 bytes) 000930 1787138444.026394 read 0x0205a2 thru 0x02f032 (0xea91 bytes) 000935 1787138444.030499 write 0x03f76e thru 0x03ffff (0x892 bytes) 000937 1787138444.032836 read 0x034e6e thru 0x037bd0 (0x2d63 bytes) 000938 1787138444.050573 mapread 0x03b867 thru 0x03ffff (0x4799 bytes) 000940 1787138444.052213 trunc from 0x040000 to 0x0131d7 (0x2ce28 bytes) 000942 1787138444.079996 trunc from 0x0131d8 to 0x033408 (0x20231 bytes) 000943 1787138444.098100 write 0x0115ff thru 0x01fd34 (0xe736 bytes) 000944 1787138444.105058 trunc from 0x033409 to 0x0171b6 (0x1c252 bytes) 000946 1787138444.131960 mapwrite 0x001177 thru 0x0012d8 (0x162 bytes) 000947 1787138444.183814 mapwrite 0x00b703 thru 0x00e57d (0x2e7b bytes) 000951 1787138444.229021 mapread 0x00a874 thru 0x00ba21 (0x11ae bytes) 000953 1787138444.230806 write 0x03c56f thru 0x03ffff (0x3a91 bytes) 000954 1787138444.233349 mapread 0x0027ca thru 0x008a84 (0x62bb bytes) 000955 1787138444.235429 read 0x034b19 thru 0x0399f7 (0x4edf bytes) 000956 1787138444.244874 write 0x03640b thru 0x036f54 (0xb4a bytes) 000959 1787138444.247843 mapread 0x031914 thru 0x039e97 (0x8584 bytes) truncating to largest ever: 0x3fd47 000960 1787138444.249188 trunc from 0x040000 to 0x03fd46 (0x2b9 bytes) 000962 1787138444.270977 read 0x006bc1 thru 0x010920 (0x9d60 bytes) 000963 1787138444.272147 trunc from 0x03fd47 to 0x0326c3 (0xd683 bytes) 000965 1787138444.290567 read 0x0148f1 thru 0x022dcc (0xe4dc bytes) 000967 1787138444.291801 mapread 0x020373 thru 0x029dcf (0x9a5d bytes) 000968 1787138444.303914 MIRROR_EXTEND to 2 mirrors 000971 1787138445.055388 write 0x0173f6 thru 0x01d758 (0x6363 bytes) 000973 1787138445.201975 trunc from 0x0326c4 to 0x02c590 (0x6133 bytes) 000974 1787138445.268781 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2c591): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c591): Mirror 2: 0x84d20499 (stale) CRC-32 checksum value for chunk [0, 0x2c591): Mirror 1: 0x151cbb2f Mirror 3: 0x151cbb2f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c591: skipped 000975 1787138446.222330 read 0x011175 thru 0x01dca5 (0xcb31 bytes) 000976 1787138446.234474 mapread 0x00c8b6 thru 0x00db10 (0x125b bytes) 000977 1787138446.245326 mapwrite 0x01df1d thru 0x023963 (0x5a47 bytes) 000979 1787138446.442326 trunc from 0x02c591 to 0x031b81 (0x55f1 bytes) 000980 1787138446.478606 mapwrite 0x01aa27 thru 0x01aa73 (0x4d bytes) 000981 1787138446.584885 mapread 0x000793 thru 0x00feb8 (0xf726 bytes) 000982 1787138446.597501 mapwrite 0x0226a3 thru 0x030f50 (0xe8ae bytes) 000983 1787138446.679150 read 0x012ad3 thru 0x0222a8 (0xf7d6 bytes) 000984 1787138446.689808 mapread 0x00e7c5 thru 0x01839d (0x9bd9 bytes) 000988 1787138446.693958 trunc from 0x031b82 to 0x0214f7 (0x1068a bytes) 000989 1787138446.725232 write 0x005746 thru 0x00b642 (0x5efd bytes) 000991 1787138446.727053 mapread 0x0158a3 thru 0x01aae5 (0x5243 bytes) 000992 1787138446.740492 MIRROR_RESYNC 3 mirrors 000994 1787138447.551333 trunc from 0x0214f8 to 0x02a462 (0x8f6b bytes) 000995 1787138447.658302 mapwrite 0x004b57 thru 0x00d6cf (0x8b79 bytes) 000996 1787138447.727078 read 0x02725e thru 0x02a462 (0x3205 bytes) 000999 1787138447.744479 read 0x02467e thru 0x0258c4 (0x1247 bytes) 001000 1787138447.763874 MIRROR_RESYNC 3 mirrors All operations completed A-OK! PASS 70a (88s) == sanity-flr test 71: check mirror extend parent fid ==== 07:20:54 (1787138454) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 07:20:58 (1787138458) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x273:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.257856 s, 8.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x270:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x273:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154934 s, 6.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x270:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x274:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x273:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x270:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x274:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x273:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x271:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 07:21:07 (1787138467) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7088864 .. Extending file size to 7986944 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7692640 .. Extending file size to 8444288 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8125728 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8542624 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9260096 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8687104 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8936160 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57214 57215 57216 57218 57221 done 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) 192.168.201.148@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 07:22:31 (1787138551) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8306880 .. Extending file size to 2676704 .. failed Extending file size to 8379616 .. Extending file size to 3155744 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6344160 .. Extending file size to 7466240 .. Extending file size to 7501248 .. Extending file size to 7509760 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8383520 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9083840 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9129568 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58662 58663 58665 58666 58668 done 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) 192.168.201.148@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (86s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 07:23:57 (1787138637) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0888805 s, 23.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.420427 s, 5.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xe255df78 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 07:24:18 (1787138658) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.1749 s, 12.0 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 07:24:26 (1787138666) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28a:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28a:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28b:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x286:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 07:24:34 (1787138674) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.172726 s, 12.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x287:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x288:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 07:24:44 (1787138684) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.263722 s, 8.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x289:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x289:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28a:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 07:24:54 (1787138694) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.274836 s, 7.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28c:0x0] } PASS 204c (37s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 07:25:30 (1787138730) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg148-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167876 s, 6.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x290:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x290:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x291:0x0] } PASS 204d (65s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 07:26:35 (1787138795) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg148-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x290:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.363925 s, 5.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x290:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x292:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x290:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x292:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (63s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 07:27:38 (1787138858) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg148-server mds-ost sync done. Creating new pool test_204f oleg148-server: Pool lustre.test_204f created Adding targets to pool oleg148-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg148-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.417828 s, 5.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x291:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x293:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x291:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x293:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg148-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg148-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg148-server: Pool lustre.test_204f destroyed PASS 204f (85s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 07:29:04 (1787138944) PASS 205a (11s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 07:29:15 (1787138955) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 07:29:26 (1787138966) oleg148-server: Pool lustre.test_206 created Adding targets to pool oleg148-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg148-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg148-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg148-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg148-server: Pool lustre.test_206 destroyed PASS 206 (24s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 07:29:49 (1787138989) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849998 s, 12.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446595 s, 23.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x725190de Mirror 3: 0x725190de PASS 207 (15s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 07:30:04 (1787139004) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 07:30:10 (1787139010) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 07:30:14 (1787139014) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.177149 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x20b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1660000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1660000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1660000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x22f0000 Read this (4 bytes): BBBB 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) PASS 209a (47s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 07:31:01 (1787139061) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0322928 s, 254 kB/s the very first read read with warm cache PASS 209b (10s) == sanity-flr test 210a: handle broken mirrored lovea ==== 07:31:11 (1787139071) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378483 s, 27.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1c6 lmm_fid: [0x200000404:0x1c6:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1c6 lmm_fid: [0x200000404:0x1c6:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29e:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834374 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-19 07:31:14.000000000 -0400 Modify: 2026-08-19 07:31:14.000000000 -0400 Change: 2026-08-19 07:31:16.000000000 -0400 Birth: 2026-08-19 07:31:13.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 07:31:22 (1787139082) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (5s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 07:31:26 (1787139086) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.828858 s, 49.4 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.155423 s, 26.4 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 19 07:31 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 19 07:31 /mnt/lustre/f211.sanity-flr PASS 211 (12s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 07:31:38 (1787139098) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (12s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 07:31:50 (1787139110) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (14s) == sanity-flr test complete, duration 1870 sec =========== 07:32:04 (1787139124) === sanity-flr: start cleanup 07:32:06 (1787139126) === === sanity-flr: finish cleanup 07:32:14 (1787139134) ===