-----============= acceptance-small: sanity-flr ============----- Fri Apr 24 11:25:06 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg154-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 11:26:06 (1777044366) === oleg154-client.virtnet: executing check_config_client /mnt/lustre oleg154-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg154-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff956345f62800.idle_timeout=debug osc.lustre-OST0001-osc-ffff956345f62800.idle_timeout=debug disable quota as required oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 11:28:10 (1777044490) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8281] == sanity-flr test 0a: lfs mirror create with -N option == 11:28:17 (1777044497) PASS 0a (55s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 11:29:13 (1777044553) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (11s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 11:29:24 (1777044564) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (10s) == sanity-flr test 0d: lfs mirror extend with -N option == 11:29:33 (1777044573) PASS 0d (81s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 11:30:55 (1777044655) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (11s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 11:31:06 (1777044666) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (9s) == sanity-flr test 0g: lfs mirror create flags support === 11:31:16 (1777044676) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff956345f62800.stats=clear osc.lustre-OST0001-osc-ffff956345f62800.stats=clear PASS 0g (23s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 11:31:38 (1777044698) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (37s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 11:32:16 (1777044736) PASS 0j (28s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 11:32:44 (1777044764) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (11s) == sanity-flr test 0m: mirror extend with -N= option ===== 11:32:56 (1777044776) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (26s) == sanity-flr test 1: create components with setstripe options ========================================================== 11:33:23 (1777044803) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (32s) == sanity-flr test 2: create components from existing files ========================================================== 11:33:55 (1777044835) PASS 2 (22s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 11:34:18 (1777044858) PASS 3 (24s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 11:34:42 (1777044882) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (22s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 11:35:05 (1777044905) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (21s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 11:35:39 (1777044939) PASS 7 (23s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 11:36:02 (1777044962) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 1.45243 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380833 s, 2.8 MB/s PASS 21 (29s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 11:36:31 (1777044991) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 1.20071 s, 6.1 MB/s PASS 22 (26s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 11:36:57 (1777045017) Stopping /mnt/lustre-ost1 (opts:) on oleg154-server oleg154-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff956345f62800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff956345f62800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg154-server: oleg154-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg154-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg154-server oleg154-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff956345f62800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff956345f62800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg154-server: oleg154-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg154-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (244s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 11:41:01 (1777045261) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 2.20282 s, 4.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg154-server oleg154-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff956345f62800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff956345f62800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg154-server: oleg154-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg154-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (130s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 11:43:11 (1777045391) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg154-server oleg154-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff956345f62800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff956345f62800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg154-server: oleg154-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg154-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg154-server oleg154-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff956345f62800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff956345f62800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg154-server: oleg154-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg154-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (257s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 11:47:28 (1777045648) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 13.0465 s, 4.0 MB/s -rw-r--r-- 1 root root 52428800 Apr 24 11:47 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m3.493s user 0m0.000s sys 0m0.120s Stopping /mnt/lustre-ost1 (opts:) on oleg154-server oleg154-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff956345f62800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff956345f62800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m3.439s user 0m0.001s sys 0m0.198s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg154-server: oleg154-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg154-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg154-server oleg154-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff956345f62800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff956345f62800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m3.710s user 0m0.001s sys 0m0.137s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg154-server: oleg154-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg154-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg154-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff956345f62800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff956345f62800.ost_server_uuid in FULL state after 0 sec PASS 33b (239s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 11:51:27 (1777045887) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (10s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 11:51:37 (1777045897) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (10s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 11:51:47 (1777045907) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (9s) == sanity-flr test 35: allow to write to mirrored files == 11:51:57 (1777045917) PASS 35 (20s) == sanity-flr test 36a: write to mirrored files ========== 11:52:18 (1777045938) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 2.99437 s, 4.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (43s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 11:53:01 (1777045981) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795861 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137258 s, 7.6 MB/s PASS 36b (25s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 11:53:27 (1777046007) PASS 36c (25s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 11:53:53 (1777046033) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00222598 s, 460 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00478218 s, 214 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00736864 s, 139 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00626564 s, 163 kB/s flv=10 olv=10 PASS 36d (35s) == sanity-flr test 37: mirror I/O API verification ======= 11:54:27 (1777046067) create test files with size 15742058 .. c69e6c9d72a24e84b58fe6281e14855e - 976d29941d4af30c6083f561729a9bba - 847089891b612d584c5d23059ad3e5b0 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Starting client: oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre PASS 37 (79s) == sanity-flr test 38: resync ============================ 11:55:46 (1777046146) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (49s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 11:56:36 (1777046196) PASS 39 (21s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 11:56:57 (1777046217) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333233 s, 3.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193895 s, 5.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3d:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (39s) == sanity-flr test 41: lfs mirror resync check =========== 11:57:36 (1777046256) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.30088 s, 7.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 1.07888 s, 3.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.714796 s, 2.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (53s) == sanity-flr test 42: lfs mirror verify ================= 11:58:29 (1777046309) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (11s) == sanity-flr test 43a: mirror pick on write ============= 11:58:41 (1777046321) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (11s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 11:58:52 (1777046332) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800861 s, 13.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (26s) == sanity-flr test 44a: lfs mirror split check =========== 11:59:19 (1777046359) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 1.28002 s, 2.5 MB/s PASS 44a (34s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 11:59:54 (1777046394) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x42:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x47:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x47:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (27s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:00:27 (1777046427) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 1.34434 s, 7.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Starting client: oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (42s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:01:10 (1777046470) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff95634a0e8800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 1.4497 s, 7.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Starting client: oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff956348175800.xattr_cache=1 PASS 44e (54s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:02:03 (1777046523) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (148s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:04:32 (1777046672) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (133s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:06:44 (1777046804) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (11s) == sanity-flr test 48: Verify snapshot mirror ============ 12:06:55 (1777046815) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.835872 s, 3.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.479995 s, 6.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xc578dfc0 (stale) Mirror 2: 0xc578dfc0 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: d4eb7c593b2f462af1a9a50c76a29e90 - mirror 1 checksum: 9c7dccae1559346111b9fdd2a5e921f1 - mirror 2 checksum: d4eb7c593b2f462af1a9a50c76a29e90 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Apr 24 12:07:03 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4c:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Apr 24 12:07:03 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: d4eb7c593b2f462af1a9a50c76a29e90 - mirror 1 checksum: 9c7dccae1559346111b9fdd2a5e921f1 - mirror 2 checksum: 9c7dccae1559346111b9fdd2a5e921f1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Apr 24 12:07:03 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4c:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Apr 24 12:07:03 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } PASS 48 (33s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 12:07:28 (1777046848) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144826 s, 7.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (25s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:07:54 (1777046874) Failing mds1 on oleg154-server Stopping /mnt/lustre-mds1 (opts:) on oleg154-server 12:08:18 (1777046898) shut down facet: mds1 facet_host: oleg154-server facet_failover_host: oleg154-server Failover mds1 to oleg154-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-MDT0000 12:09:12 (1777046952) targets are mounted 12:09:12 (1777046952) facet_failover done oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166822 s, 6.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } PASS 50A (124s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:09:59 (1777046999) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.208429 s, 98.3 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0787548 s, 260 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (32s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:10:32 (1777047032) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.270753 s, 75.6 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0779618 s, 263 kB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (36s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:11:07 (1777047067) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0569774 s, 288 kB/s PASS 50c (24s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:11:32 (1777047092) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.256935 s, 79.7 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (27s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:12:00 (1777047120) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0410382 s, 99.8 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0787921 s, 52.0 kB/s PASS 60a (26s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:12:25 (1777047145) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) PASS 60b (80s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:13:46 (1777047226) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (58s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:14:43 (1777047283) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (40s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:15:24 (1777047324) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (43s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:16:08 (1777047368) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (24s) == sanity-flr test 70: mirror create and split race ====== 12:16:31 (1777047391) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 56906 mirror split pid 56908 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4993: 56906 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4993: 56908 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (82s) == sanity-flr test 70a: flr mode fsx test ================ 12:17:53 (1777047473) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 7480 truncating to largest ever: 0x3838f 000002 1777047481.071694 trunc from 00000000 to 0x03838e (0x3838f bytes) 000003 1777047481.146764 trunc from 0x003bd9 to 0x0062ed (0x2715 bytes) 000004 1777047481.187053 mapread 0x01f2f8 thru 0x02646c (0x7175 bytes) 000005 1777047481.304258 fallocate 0x0293f0 thru 0x02e830 (0x5441 bytes) 000006 1777047481.334986 punch from 0x02a638 to 0x02b13c (0xb05 bytes) 000008 1777047481.354611 mapwrite 0x004120 thru 0x00c19f (0x8080 bytes) 000009 1777047481.551610 read 0x02fa67 thru 0x0317c4 (0x1d5e bytes) 000010 1777047481.585672 punch from 0x007dcb to 0x013c70 (0xbea6 bytes) 000011 1777047481.612051 mapwrite 0x03831e thru 0x03b591 (0x3274 bytes) 000012 1777047481.766071 fallocate 0x01d2ee thru 0x02208f (0x4da2 bytes) 000013 1777047481.830485 mapwrite 0x0003cd thru 0x009255 (0x8e89 bytes) 000014 1777047482.032480 trunc from 0x037c5d to 0x03b591 (0x3935 bytes) 000015 1777047482.051512 fallocate 0x0198e8 thru 0x02342d (0x9b46 bytes) 000016 1777047482.072792 mapread 0x02bcf5 thru 0x02df62 (0x226e bytes) 000017 1777047482.139302 write 0x000eb6 thru 0x0035da (0x2725 bytes) truncating to largest ever: 0x3f06d 000018 1777047482.154322 trunc from 0x03b592 to 0x03f06c (0x3adb bytes) 000019 1777047482.223699 punch from 0x01fb7d to 0x02dd4a (0xe1ce bytes) 000020 1777047482.250202 fallocate 0x0375b4 thru 0x03dcc2 (0x670f bytes) 000022 1777047482.304587 trunc from 0x0354c0 to 0x03ab69 (0x56aa bytes) 000023 1777047482.342292 MIRROR_EXTEND to 2 mirrors 000025 1777047484.802334 punch from 0x00ba11 to 0x015fce (0xa5be bytes) 000026 1777047484.886603 trunc from 0x03f06d to 0x01b833 (0x23839 bytes) 000027 1777047485.038361 mapwrite 0x024393 thru 0x03342b (0xf099 bytes) 000028 1777047485.321109 read 0x017ffe thru 0x01cacd (0x4ad0 bytes) 000029 1777047485.419017 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3342c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3342c): Mirror 2: 0xdab2986a (stale) CRC-32 checksum value for chunk [0, 0x3342c): Mirror 1: 0x7923c5d5 Mirror 3: 0x7923c5d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3342c: skipped 000030 1777047489.131516 punch from 0x0214fe to 0x02b192 (0x9c95 bytes) 000031 1777047489.305028 write 0x01076f thru 0x01ffae (0xf840 bytes) 000032 1777047489.373713 mapread 0x01af07 thru 0x02827f (0xd379 bytes) 000033 1777047489.421582 read 0x029477 thru 0x03342b (0x9fb5 bytes) 000034 1777047489.476552 trunc from 0x03342c to 0x013f14 (0x1f517 bytes) 000035 1777047489.548691 trunc from 0x0087ae to 0x013f14 (0xb767 bytes) 000036 1777047489.632262 read 0x011a07 thru 0x013f14 (0x250e bytes) 000037 1777047489.691552 punch from 0x007d1b to 0x00fc97 (0x7f7d bytes) 000038 1777047489.776934 MIRROR_RESYNC 3 mirrors 000040 1777047493.379570 punch from 0x001e6c to 0x011654 (0xf7e9 bytes) 000041 1777047493.528487 mapwrite 0x03c744 thru 0x03ffff (0x38bc bytes) 000043 1777047493.891474 trunc from 0x007f78 to 0x00b4c0 (0x3549 bytes) 000044 1777047493.939143 write 0x002ebd thru 0x011ecf (0xf013 bytes) 000046 1777047494.094251 MIRROR_RESYNC 3 mirrors 000047 1777047497.217002 read 0x017759 thru 0x026a20 (0xf2c8 bytes) 000048 1777047497.270205 mapread 0x02169c thru 0x02f647 (0xdfac bytes) 000049 1777047497.343605 mapread 0x02b040 thru 0x034794 (0x9755 bytes) 000050 1777047497.409433 mapread 0x023abf thru 0x026405 (0x2947 bytes) 000051 1777047497.423480 trunc from 0x040000 to 0x013d03 (0x2c2fc bytes) 000052 1777047497.708185 mapread 0x013b8c thru 0x013d03 (0x178 bytes) 000053 1777047497.793500 trunc from 0x009110 to 0x010712 (0x7603 bytes) 000054 1777047497.817418 write 0x00002c thru 0x00c2e8 (0xc2bd bytes) 000055 1777047497.890324 mapwrite 0x0347b5 thru 0x03c9bd (0x8209 bytes) 000056 1777047498.214224 read 0x001f44 thru 0x00e1f6 (0xc2b3 bytes) 000058 1777047498.270705 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3c9be): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c9be): Mirror 2: 0xb55b664f (stale) Mirror 3: 0xb55b664f (stale) CRC-32 checksum value for chunk [0, 0x3c9be): Mirror 1: 0x23ed7e7 Mirror 4: 0x23ed7e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c9be: skipped 000059 1777047501.712352 trunc from 0x016a75 to 0x019609 (0x2b95 bytes) 000060 1777047501.892970 punch from 0x00f283 to 0x01b269 (0xbfe7 bytes) 000061 1777047501.918401 trunc from 0x021629 to 0x028e03 (0x77db bytes) fallocating to largest ever: 0x40000 000062 1777047501.943624 fallocate 0x03a792 thru 0x03ffff (0x586e bytes) 000063 1777047501.973493 punch from 0x0219f2 to 0x02f49d (0xdaac bytes) 000065 1777047502.005478 trunc from 0x03c9be to 0x03748b (0x5532 bytes) 000066 1777047502.095157 mapwrite 0x03df6b thru 0x03ffff (0x2095 bytes) 000067 1777047502.374004 trunc from 0x040000 to 0x00ba46 (0x345b9 bytes) 000068 1777047502.460425 trunc from 0x00ba47 to 0x00bc97 (0x251 bytes) 000069 1777047502.593483 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xbc98): Mirror 3: 0xcbe35967 (stale) Mirror 4: 0x7e6c4c12 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbc98): Mirror 3: 0xcbe35967 (stale) Mirror 4: 0x7e6c4c12 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbc98: skipped 000070 1777047503.776897 punch from 0x005263 to 0x00bc97 (0x6a35 bytes) 000071 1777047503.817844 trunc from 0x00bc98 to 0x02784c (0x1bbb5 bytes) 000072 1777047503.895818 fallocate 0x02b6ea thru 0x02bae7 (0x3fe bytes) 000073 1777047503.940374 mapwrite 0x03f767 thru 0x03ffff (0x899 bytes) 000074 1777047504.187166 punch from 0x003fb2 to 0x00dbfd (0x9c4c bytes) 000075 1777047504.228907 trunc from 0x040000 to 0x0093c5 (0x36c3a bytes) 000076 1777047504.324517 read 0x0016da thru 0x00712b (0x5a52 bytes) 000077 1777047504.423571 write 0x01c631 thru 0x0229bb (0x638b bytes) 000078 1777047504.434394 mapread 0x002db7 thru 0x00ed80 (0xbfca bytes) 000079 1777047504.534285 mapwrite 0x01e237 thru 0x02323f (0x5009 bytes) 000081 1777047504.806845 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x23240): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23240): Mirror 3: 0xdd564705 (stale) Mirror 4: 0xbd7575c3 (stale) CRC-32 checksum value for chunk [0, 0x23240): Mirror 1: 0x1ea92969 Mirror 5: 0x1ea92969 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23240: skipped 000082 1777047509.213444 read 0x00ddf9 thru 0x00ea36 (0xc3e bytes) 000083 1777047509.300991 MIRROR_RESYNC 4 mirrors 000084 1777047512.090539 mapwrite 0x024de9 thru 0x02acdb (0x5ef3 bytes) 000085 1777047512.445301 trunc from 0x02acdc to 0x03cc0d (0x11f32 bytes) 000086 1777047512.589362 mapread 0x02d42b thru 0x038588 (0xb15e bytes) 000087 1777047512.720613 trunc from 0x0373b6 to 0x03cc0d (0x5858 bytes) 000088 1777047512.792331 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3cc0e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cc0e): Mirror 3: 0x5c4b5ab7 (stale) Mirror 4: 0x5c4b5ab7 (stale) Mirror 5: 0x5c4b5ab7 (stale) CRC-32 checksum value for chunk [0, 0x3cc0e): Mirror 1: 0x46edeee5 Mirror 6: 0x46edeee5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cc0e: skipped 000089 1777047516.506109 mapwrite 0x0136bf thru 0x01c6e4 (0x9026 bytes) 000091 1777047516.879975 read 0x0371f3 thru 0x03cc0d (0x5a1b bytes) 000092 1777047516.919496 mapread 0x031f2b thru 0x03b9bc (0x9a92 bytes) 000093 1777047516.959214 punch from 0x03b9ca to 0x03cc0d (0x1244 bytes) 000094 1777047516.986114 read 0x00b0d1 thru 0x010bc5 (0x5af5 bytes) 000095 1777047517.043765 write 0x00d25c thru 0x010ae1 (0x3886 bytes) 000096 1777047517.069313 trunc from 0x001d31 to 0x00df52 (0xc222 bytes) 000097 1777047517.150236 read 0x022e11 thru 0x02e97a (0xbb6a bytes) 000098 1777047517.159759 punch from 0x01c34e to 0x01de70 (0x1b23 bytes) 000099 1777047517.202990 fallocate 0x03e3fa thru 0x03ffff (0x1c06 bytes) 000100 1777047517.246120 read 0x015c67 thru 0x02364d (0xd9e7 bytes) 000101 1777047517.302862 read 0x03461b thru 0x03ffff (0xb9e5 bytes) 000102 1777047517.320981 read 0x03aff3 thru 0x03ffff (0x500d bytes) 000103 1777047517.323316 trunc from 0x040000 to 0x03c954 (0x36ab bytes) 000104 1777047517.425291 punch from 0x029a4d to 0x039446 (0xf9fa bytes) 000105 1777047517.453260 mapread 0x0216f3 thru 0x02f067 (0xd975 bytes) 000108 1777047517.519594 mapwrite 0x01e1a8 thru 0x020752 (0x25ab bytes) 000109 1777047517.594463 fallocate 0x03f00e thru 0x03ffff (0xff2 bytes) 000110 1777047517.613383 punch from 0x00c226 to 0x010044 (0x3e1f bytes) 000111 1777047517.631237 trunc from 0x03c955 to 0x02e889 (0xe0cb bytes) 000112 1777047517.683130 trunc from 0x02e88a to 0x015456 (0x19433 bytes) 000113 1777047517.754628 mapwrite 0x009e8a thru 0x018c89 (0xee00 bytes) 000114 1777047518.362164 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18c8a): Mirror 3: 0xca04cc04 (stale) Mirror 4: 0xca04cc04 (stale) Mirror 5: 0xca04cc04 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18c8a): Mirror 3: 0xca04cc04 (stale) Mirror 4: 0xca04cc04 (stale) Mirror 5: 0xca04cc04 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18c8a: skipped 000115 1777047519.392651 mapread 0x0170b1 thru 0x018c89 (0x1bd9 bytes) 000116 1777047519.445643 trunc from 0x018c8a to 0x02c725 (0x13a9c bytes) 000117 1777047519.539085 fallocate 0x039b0e thru 0x03e287 (0x477a bytes) 000118 1777047519.569493 mapread 0x0319b0 thru 0x031ad1 (0x122 bytes) 000119 1777047519.633933 mapwrite 0x031e71 thru 0x03430b (0x249b bytes) 000120 1777047519.773837 mapread 0x034c82 thru 0x03ded5 (0x9254 bytes) 000121 1777047519.816470 punch from 0x022f06 to 0x02624c (0x3347 bytes) 000122 1777047519.830886 mapwrite 0x016ee9 thru 0x01ae77 (0x3f8f bytes) 000123 1777047519.951444 trunc from 0x03e288 to 0x018700 (0x25b87 bytes) 000124 1777047520.016516 trunc from 0x018701 to 0x0118a8 (0x6e58 bytes) 000125 1777047520.093464 mapread 0x001f5d thru 0x00975d (0x7801 bytes) 000126 1777047520.234251 write 0x005991 thru 0x012264 (0xc8d4 bytes) 000127 1777047520.259019 mapwrite 0x037352 thru 0x03a5f4 (0x32a3 bytes) 000129 1777047520.496839 write 0x034cc0 thru 0x038cb6 (0x3ff7 bytes) 000130 1777047520.510234 punch from 0x009964 to 0x0144ed (0xab8a bytes) 000131 1777047520.543525 read 0x02ebb9 thru 0x03a5f4 (0xba3c bytes) 000133 1777047520.580384 punch from 0x035db9 to 0x03a5f4 (0x483c bytes) 000134 1777047520.628695 read 0x023fe0 thru 0x030081 (0xc0a2 bytes) 000135 1777047520.697565 read 0x02faba thru 0x034525 (0x4a6c bytes) 000136 1777047520.726673 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a5f5): Mirror 3: 0x6d59e7be (stale) Mirror 4: 0x6d59e7be (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a5f5): Mirror 3: 0x6d59e7be (stale) Mirror 4: 0x6d59e7be (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a5f5: skipped 000137 1777047521.982651 mapread 0x00e97d thru 0x01bba2 (0xd226 bytes) 000138 1777047522.129654 mapwrite 0x01e6d9 thru 0x02c3b0 (0xdcd8 bytes) 000139 1777047522.332096 read 0x010fca thru 0x0166af (0x56e6 bytes) 000140 1777047522.380391 MIRROR_RESYNC 3 mirrors 000141 1777047527.165594 read 0x022b9b thru 0x02e01f (0xb485 bytes) 000142 1777047527.228443 punch from 0x01dc9d to 0x01fc77 (0x1fdb bytes) 000143 1777047527.532581 fallocate 0x0129ee thru 0x019653 (0x6c66 bytes) 000144 1777047527.576273 mapwrite 0x01be08 thru 0x0207dd (0x49d6 bytes) 000145 1777047527.839329 trunc from 0x03a5f5 to 0x02df57 (0xc69d bytes) 000147 1777047527.949991 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2df58): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2df58): Mirror 3: 0xcf060524 (stale) Mirror 4: 0xcf060524 (stale) CRC-32 checksum value for chunk [0, 0x2df58): Mirror 1: 0xc55780f2 Mirror 5: 0xc55780f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2df58: skipped 000148 1777047531.747319 write 0x026139 thru 0x032b8a (0xca52 bytes) 000149 1777047532.061101 fallocate 0x02eece thru 0x03729e (0x83d1 bytes) 000150 1777047532.075635 read 0x018e89 thru 0x01a229 (0x13a1 bytes) 000151 1777047532.111025 fallocate 0x00a702 thru 0x010017 (0x5916 bytes) 000152 1777047532.139066 punch from 0x0188b3 to 0x0231a7 (0xa8f5 bytes) 000153 1777047532.157183 mapwrite 0x029bb9 thru 0x02e408 (0x4850 bytes) 000154 1777047532.236571 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32b8b): Mirror 3: 0x280e102d (stale) Mirror 5: 0x3aa05aa8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32b8b): Mirror 3: 0x280e102d (stale) Mirror 5: 0x3aa05aa8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32b8b: skipped 000155 1777047533.622308 fallocate 0x024339 thru 0x02770e (0x33d6 bytes) 000156 1777047533.653775 punch from 0x0088c7 to 0x016e16 (0xe550 bytes) 000158 1777047533.695770 fallocate 0x01984d thru 0x026381 (0xcb35 bytes) 000159 1777047533.711411 fallocate 0x03d283 thru 0x03ffff (0x2d7d bytes) 000160 1777047533.735067 mapread 0x037915 thru 0x038338 (0xa24 bytes) 000161 1777047533.786182 fallocate 0x03f30e thru 0x03ffff (0xcf2 bytes) 000162 1777047533.820063 read 0x01ea42 thru 0x02caac (0xe06b bytes) 000163 1777047533.877333 trunc from 0x03d984 to 0x03ffff (0x267c bytes) 000164 1777047533.907810 trunc from 0x040000 to 0x01de15 (0x221ea bytes) 000165 1777047534.014124 trunc from 0x016997 to 0x01de15 (0x747f bytes) 000166 1777047534.038966 punch from 0x0011fb to 0x00edac (0xdbb2 bytes) 000167 1777047534.064323 mapread 0x01b083 thru 0x01de15 (0x2d93 bytes) 000168 1777047534.119861 MIRROR_RESYNC 3 mirrors 000169 1777047536.654884 mapwrite 0x01e44b thru 0x029fe4 (0xbb9a bytes) 000171 1777047537.181583 mapread 0x01a5f5 thru 0x0259d7 (0xb3e3 bytes) 000173 1777047537.194297 punch from 0x0040e8 to 0x00bd44 (0x7c5d bytes) 000174 1777047537.227605 mapwrite 0x0180ca thru 0x01bd9f (0x3cd6 bytes) 000176 1777047537.371691 MIRROR_RESYNC 3 mirrors 000177 1777047539.769551 trunc from 0x029fe5 to 0x037868 (0xd884 bytes) 000178 1777047540.000686 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x37869): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37869): Mirror 3: 0x7e6bfba5 (stale) Mirror 5: 0x7e6bfba5 (stale) CRC-32 checksum value for chunk [0, 0x37869): Mirror 1: 0x7e6bfba5 Mirror 6: 0x7e6bfba5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37869: skipped 000179 1777047543.571990 read 0x019a1b thru 0x01dd0a (0x42f0 bytes) 000180 1777047543.671137 mapread 0x018ba6 thru 0x02888d (0xfce8 bytes) 000181 1777047543.896141 punch from 0x02e520 to 0x037868 (0x9349 bytes) 000182 1777047544.075462 trunc from 0x037869 to 0x023fe4 (0x13884 bytes) 000183 1777047544.163433 punch from 0x021c1a to 0x023a8c (0x1e73 bytes) 000184 1777047544.189385 trunc from 0x002408 to 0x002e7c (0xa75 bytes) 000185 1777047544.241609 MIRROR_RESYNC 4 mirrors 000186 1777047549.074458 punch from 0x01b041 to 0x023cf8 (0x8cb8 bytes) 000187 1777047549.234427 trunc from 0x00fd06 to 0x0115d9 (0x18d4 bytes) 000188 1777047549.259101 mapwrite 0x01c124 thru 0x0257e2 (0x96bf bytes) 000189 1777047549.573471 trunc from 0x02411b to 0x0257e2 (0x16c8 bytes) 000190 1777047549.597811 write 0x0018ed thru 0x0115a4 (0xfcb8 bytes) 000191 1777047549.660493 mapread 0x01aa42 thru 0x01abb5 (0x174 bytes) 000192 1777047549.683095 mapwrite 0x0204b4 thru 0x0268e2 (0x642f bytes) 000193 1777047550.028815 write 0x00f5a2 thru 0x01868e (0x90ed bytes) 000194 1777047550.063054 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x268e3): Mirror 5: 0x6820f60a (stale) Mirror 6: 0x6820f60a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x268e3): Mirror 5: 0x6820f60a (stale) Mirror 6: 0x6820f60a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x268e3: skipped 000195 1777047551.569730 mapwrite 0x01dbf0 thru 0x021d44 (0x4155 bytes) 000196 1777047551.799093 mapwrite 0x03517a thru 0x0354de (0x365 bytes) 000197 1777047552.075651 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x354df): Mirror 6: 0x91ff4ec3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x354df): Mirror 6: 0x91ff4ec3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x354df: skipped 000198 1777047553.139961 trunc from 0x0354df to 0x02d47b (0x8063 bytes) 000199 1777047553.225106 read 0x000b0c thru 0x007ad4 (0x6fc9 bytes) 000200 1777047553.261123 fallocate 0x039062 thru 0x03ffff (0x6f9e bytes) 000201 1777047553.335052 MIRROR_RESYNC 2 mirrors 000202 1777047555.541479 mapread 0x007091 thru 0x00df25 (0x6e95 bytes) 000203 1777047555.656081 mapread 0x03882c thru 0x0398fc (0x10d1 bytes) 000204 1777047555.707298 mapread 0x01dccc thru 0x024045 (0x637a bytes) 000205 1777047555.792459 punch from 0x01c428 to 0x02a28e (0xde67 bytes) 000206 1777047555.930801 mapread 0x03713b thru 0x03ffff (0x8ec5 bytes) 000207 1777047555.990033 punch from 0x028c31 to 0x02c602 (0x39d2 bytes) 000208 1777047556.011865 trunc from 0x012d54 to 0x01ad97 (0x8044 bytes) 000209 1777047556.052984 MIRROR_EXTEND to 3 mirrors