-----============= acceptance-small: sanity-flr ============----- Thu Aug 6 20:33:29 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg405-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 20:33:52 (1786062832) === oleg405-client.virtnet: executing check_config_client /mnt/lustre oleg405-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg405-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8ad9886d7000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8ad9886d7000.idle_timeout=debug disable quota as required oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 20:34:43 (1786062883) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8243] == sanity-flr test 0a: lfs mirror create with -N option == 20:34:46 (1786062886) PASS 0a (22s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:35:08 (1786062908) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (6s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:35:14 (1786062914) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (5s) == sanity-flr test 0d: lfs mirror extend with -N option == 20:35:19 (1786062919) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:35:45 (1786062945) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:35:49 (1786062949) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (5s) == sanity-flr test 0g: lfs mirror create flags support === 20:35:54 (1786062954) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8ad9886d7000.stats=clear osc.lustre-OST0001-osc-ffff8ad9886d7000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:36:04 (1786062964) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:36:15 (1786062975) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:36:25 (1786062985) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (5s) == sanity-flr test 0m: mirror extend with -N= option ===== 20:36:31 (1786062991) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:36:42 (1786063002) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 20:36:54 (1786063014) PASS 2 (10s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:37:03 (1786063023) PASS 3 (10s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:37:13 (1786063033) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (10s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:37:23 (1786063043) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (10s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:37:38 (1786063058) PASS 7 (9s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:37:47 (1786063067) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 1.68306 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1587 s, 6.6 MB/s PASS 21 (13s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:38:00 (1786063080) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.644805 s, 21.1 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 20:38:11 (1786063091) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (15s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:38:26 (1786063106) Stopping /mnt/lustre-ost1 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ad9886d7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ad9886d7000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg405-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ad9886d7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ad9886d7000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg405-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (96s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:40:01 (1786063201) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.489872 s, 15.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ad9886d7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ad9886d7000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg405-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (51s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:40:52 (1786063252) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ad9886d7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ad9886d7000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg405-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ad9886d7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ad9886d7000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg405-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (100s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:42:33 (1786063353) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.66595 s, 19.7 MB/s -rw-r--r-- 1 root root 52428800 Aug 6 20:42 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.614s user 0m0.000s sys 0m0.184s Stopping /mnt/lustre-ost1 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ad9886d7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ad9886d7000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.540s user 0m0.001s sys 0m0.178s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg405-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ad9886d7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ad9886d7000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.460s user 0m0.002s sys 0m0.197s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg405-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg405-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ad9886d7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ad9886d7000.ost_server_uuid in FULL state after 0 sec PASS 33b (114s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:44:27 (1786063467) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (5s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:44:32 (1786063472) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (5s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:44:37 (1786063477) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (5s) == sanity-flr test 35: allow to write to mirrored files == 20:44:42 (1786063482) PASS 35 (10s) == sanity-flr test 36a: write to mirrored files ========== 20:44:52 (1786063492) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.767822 s, 16.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (18s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:45:10 (1786063510) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403803 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339981 s, 30.8 MB/s PASS 36b (10s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:45:21 (1786063521) PASS 36c (12s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:45:33 (1786063533) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00165487 s, 619 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00483238 s, 212 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00251706 s, 407 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0034889 s, 294 kB/s flv=10 olv=10 PASS 36d (14s) == sanity-flr test 37: mirror I/O API verification ======= 20:45:47 (1786063547) create test files with size 15744870 .. 5c1d4b8e2b1cee1c247165f7997b8205 - 6b9d27b775f11afe59a6eb5e151e61bf - a2e60a64785ee2e5a07c77dc6d742d3b - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg405-client.virtnet /mnt/lustre (opts:) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre PASS 37 (28s) == sanity-flr test 38: resync ============================ 20:46:15 (1786063575) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:46:32 (1786063592) PASS 39 (10s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:46:42 (1786063602) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06065 s, 17.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303116 s, 34.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (12s) == sanity-flr test 41: lfs mirror resync check =========== 20:46:54 (1786063614) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0725668 s, 28.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.266727 s, 15.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.155576 s, 13.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 20:47:14 (1786063634) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (5s) == sanity-flr test 43a: mirror pick on write ============= 20:47:19 (1786063639) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (5s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:47:24 (1786063644) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273063 s, 38.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (11s) == sanity-flr test 44a: lfs mirror split check =========== 20:47:35 (1786063655) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.300226 s, 10.5 MB/s PASS 44a (13s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:47:48 (1786063668) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:48:01 (1786063681) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.396677 s, 26.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg405-client.virtnet /mnt/lustre (opts:) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (17s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:48:18 (1786063698) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8ad991243800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.527575 s, 19.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg405-client.virtnet /mnt/lustre (opts:) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8ad98618a800.xattr_cache=1 PASS 44e (22s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:48:41 (1786063721) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (48s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:49:29 (1786063769) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (53s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:50:22 (1786063822) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 20:50:26 (1786063826) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.228417 s, 13.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.184536 s, 17.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x8ff7cdc8 (nosync,stale) Mirror 2: 0x8ff7cdc8 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 99a593ae86e67bb8e6b90ab61ed497a4 - mirror 1 checksum: 49b2c809b42770cf67ba9aad9dac7f06 - mirror 2 checksum: 99a593ae86e67bb8e6b90ab61ed497a4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-06T20:50:31-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbe:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-06T20:50:31-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 99a593ae86e67bb8e6b90ab61ed497a4 - mirror 1 checksum: 49b2c809b42770cf67ba9aad9dac7f06 - mirror 2 checksum: 49b2c809b42770cf67ba9aad9dac7f06 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-06T20:50:31-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbe:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-06T20:50:31-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbf:0x0] } PASS 48 (14s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 20:50:40 (1786063840) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324276 s, 32.3 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:50:50 (1786063850) Failing mds1 on oleg405-server Stopping /mnt/lustre-mds1 (opts:) on oleg405-server 20:51:03 (1786063863) shut down facet: mds1 facet_host: oleg405-server facet_failover_host: oleg405-server Failover mds1 to oleg405-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-MDT0000 20:51:38 (1786063898) targets are mounted 20:51:38 (1786063898) facet_failover done oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479506 s, 21.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } PASS 50A (69s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:52:00 (1786063920) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0583602 s, 351 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0158741 s, 1.3 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (13s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:52:13 (1786063933) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0870649 s, 235 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0143913 s, 1.4 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (12s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:52:26 (1786063946) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00651755 s, 2.5 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:52:35 (1786063955) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0770464 s, 266 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (11s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:52:46 (1786063966) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0167657 s, 244 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.028805 s, 142 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:52:56 (1786063976) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (29s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:53:26 (1786064006) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (34s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:54:00 (1786064040) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (27s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:54:27 (1786064067) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:54:54 (1786064094) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (13s) == sanity-flr test 70: mirror create and split race ====== 20:55:07 (1786064107) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 57610 mirror split pid 57612 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57610 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57612 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 20:56:16 (1786064176) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 4179 000002 1786064179.704547 mapwrite 0x00b2d8 thru 0x00d2f7 (0x2020 bytes) zero_range to largest ever: 0x9503 000003 1786064179.780546 trunc from 0x0080b4 to 0x009502 (0x144f bytes) 000004 1786064179.787410 mapread 0x002ba6 thru 0x00d2f7 (0xa752 bytes) 000005 1786064179.807681 MIRROR_EXTEND to 2 mirrors fallocating to largest ever: 0x29f75 000006 1786064180.596873 fallocate 0x0237fb thru 0x029f74 (0x677a bytes) 000007 1786064180.661119 trunc from 0x029f75 to 0x005450 (0x24b24 bytes) 000008 1786064180.706668 trunc from 0x001f52 to 0x005450 (0x34ff bytes) 000009 1786064180.723569 mapwrite 0x0328eb thru 0x034111 (0x1827 bytes) 000010 1786064180.831713 punch from 0x028586 to 0x02c554 (0x3fcf bytes) 000011 1786064180.849175 punch from 0x017567 to 0x01a8c0 (0x335a bytes) 000012 1786064180.868844 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) 000013 1786064181.143933 write 0x035f0d thru 0x03ffff (0xa0f3 bytes) 000015 1786064181.156793 trunc from 0x018b0d to 0x026a4d (0xdf41 bytes) 000016 1786064181.163268 write 0x0260d6 thru 0x02ad92 (0x4cbd bytes) 000017 1786064181.176648 write 0x030b62 thru 0x03c042 (0xb4e1 bytes) 000018 1786064181.185074 write 0x0388d4 thru 0x03ffff (0x772c bytes) 000020 1786064181.188080 trunc from 0x03a677 to 0x03ffff (0x5989 bytes) 000021 1786064181.208241 fallocate 0x01a2fb thru 0x01d44f (0x3155 bytes) 000022 1786064181.217847 read 0x02ec0c thru 0x03c7f3 (0xdbe8 bytes) 000023 1786064181.228160 mapread 0x03d0fb thru 0x03ffff (0x2f05 bytes) 000024 1786064181.248604 fallocate 0x005cc9 thru 0x0132c3 (0xd5fb bytes) 000025 1786064181.261624 trunc from 0x00b5a9 to 0x011224 (0x5c7c bytes) 000026 1786064181.275833 fallocate 0x00db9e thru 0x019679 (0xbadc bytes) 000027 1786064181.297677 MIRROR_EXTEND to 2 mirrors 000028 1786064182.168891 fallocate 0x01eeb4 thru 0x02112b (0x2278 bytes) 000029 1786064182.225118 mapwrite 0x01a73a thru 0x023f3b (0x9802 bytes) fallocating to largest ever: 0x2b387 000030 1786064182.270511 fallocate 0x01e7af thru 0x02b386 (0xcbd8 bytes) 000031 1786064182.276900 fallocate 0x024a4b thru 0x02a8ef (0x5ea5 bytes) 000032 1786064182.290614 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x792d55ab (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x84939919 Mirror 3: 0x84939919 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000033 1786064183.216563 write 0x02f9d1 thru 0x0305f8 (0xc28 bytes) 000034 1786064183.279095 trunc from 0x00dd02 to 0x019022 (0xb321 bytes) 000035 1786064183.284978 punch from 0x03f8b7 to 0x03ffff (0x749 bytes) 000036 1786064183.290805 mapread 0x012e48 thru 0x01d90d (0xaac6 bytes) 000037 1786064183.326619 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x84939919 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x84939919 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000038 1786064183.702653 trunc from 0x040000 to 0x02ad83 (0x1527c bytes) 000039 1786064183.769460 write 0x025def thru 0x02de6f (0x8081 bytes) 000040 1786064183.783287 mapwrite 0x01abec thru 0x023871 (0x8c86 bytes) 000041 1786064183.851541 mapwrite 0x013092 thru 0x021e45 (0xedb4 bytes) 000042 1786064183.904433 mapwrite 0x037f84 thru 0x03aa2c (0x2aa9 bytes) 000043 1786064183.957959 mapread 0x02dd86 thru 0x03921c (0xb497 bytes) 000044 1786064183.967702 punch from 0x014c94 to 0x02306f (0xe3dc bytes) 000045 1786064183.973928 trunc from 0x0132ea to 0x0220d3 (0xedea bytes) 000046 1786064183.979794 read 0x011dbe thru 0x014737 (0x297a bytes) 000047 1786064183.987558 read 0x012c54 thru 0x01b929 (0x8cd6 bytes) 000048 1786064184.023674 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) 000049 1786064184.231499 read 0x031b4d thru 0x03aa2c (0x8ee0 bytes) 000050 1786064184.250647 write 0x002f05 thru 0x00e8cd (0xb9c9 bytes) 000052 1786064184.270823 punch from 0x025eef to 0x034cc6 (0xedd8 bytes) 000053 1786064184.278947 write 0x005010 thru 0x00545a (0x44b bytes) 000056 1786064184.282734 mapread 0x02092f thru 0x025507 (0x4bd9 bytes) 000057 1786064184.320227 MIRROR_EXTEND to 2 mirrors 000060 1786064185.183434 MIRROR_EXTEND to 3 mirrors 000061 1786064186.145706 trunc from 0x024c3f to 0x03137c (0xc73e bytes) 000062 1786064186.198967 write 0x028919 thru 0x036af0 (0xe1d8 bytes) 000063 1786064186.216686 fallocate 0x005fd9 thru 0x0145da (0xe602 bytes) 000064 1786064186.222871 trunc from 0x003444 to 0x0034c5 (0x82 bytes) 000066 1786064186.228574 fallocate 0x0086b3 thru 0x016f49 (0xe897 bytes) 000067 1786064186.233758 punch from 0x01c5e9 to 0x024082 (0x7a9a bytes) 000068 1786064186.239793 mapwrite 0x03d462 thru 0x03ffff (0x2b9e bytes) 000069 1786064186.301737 write 0x005304 thru 0x008c75 (0x3972 bytes) 000070 1786064186.314762 punch from 0x024f83 to 0x02e4de (0x955c bytes) 000071 1786064186.325544 MIRROR_RESYNC 3 mirrors 000072 1786064187.544450 mapread 0x01cbdc thru 0x01e497 (0x18bc bytes) 000075 1786064187.574859 MIRROR_EXTEND to 4 mirrors 000076 1786064188.431573 write 0x00b49d thru 0x011626 (0x618a bytes) 000077 1786064188.491633 write 0x014e30 thru 0x01df44 (0x9115 bytes) 000078 1786064188.508274 write 0x038e34 thru 0x03ffff (0x71cc bytes) zero_range to largest ever: 0x33190 000080 1786064188.518006 trunc from 0x0300cd to 0x03318f (0x30c3 bytes) 000081 1786064188.526549 trunc from 0x016e16 to 0x01b601 (0x47ec bytes) 000082 1786064188.551046 mapwrite 0x023dbd thru 0x02eaa1 (0xace5 bytes) 000083 1786064188.612398 read 0x00b16e thru 0x01a6bd (0xf550 bytes) 000084 1786064188.625922 write 0x01d3f5 thru 0x02882f (0xb43b bytes) 000085 1786064188.633007 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x52a1807a (stale) Mirror 4: 0x52a1807a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x52a1807a (stale) Mirror 4: 0x52a1807a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000086 1786064189.113086 punch from 0x028d4b to 0x02aa68 (0x1d1e bytes) 000087 1786064189.120430 write 0x02c0d1 thru 0x037cb6 (0xbbe6 bytes) 000089 1786064189.140145 mapread 0x03e60b thru 0x03ffff (0x19f5 bytes) 000090 1786064189.152597 trunc from 0x040000 to 0x01f9ab (0x20654 bytes) 000091 1786064189.173759 punch from 0x01d898 to 0x01db4f (0x2b8 bytes) 000092 1786064189.182765 write 0x03e9b4 thru 0x03ffff (0x164c bytes) 000093 1786064189.188118 mapread 0x033e35 thru 0x03ffff (0xc1cb bytes) 000094 1786064189.225130 trunc from 0x00649f to 0x00823a (0x1d9c bytes) 000095 1786064189.252402 punch from 0x00a797 to 0x00ba8a (0x12f4 bytes) 000096 1786064189.289815 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x52a1807a (stale) Mirror 4: 0x52a1807a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x272ff10b Mirror 5: 0x272ff10b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000098 1786064190.933840 trunc from 0x040000 to 0x00d65d (0x329a2 bytes) 000100 1786064191.033001 read 0x0053b2 thru 0x00d65d (0x82ac bytes) 000101 1786064191.050389 mapwrite 0x015c78 thru 0x024db8 (0xf141 bytes) 000102 1786064191.221176 mapwrite 0x003ced thru 0x006ca9 (0x2fbd bytes) 000103 1786064191.283059 punch from 0x016ed0 to 0x01fd69 (0x8e9a bytes) 000104 1786064191.296035 mapread 0x007fac thru 0x00d933 (0x5988 bytes) 000105 1786064191.298961 write 0x02bc1e thru 0x033d37 (0x811a bytes) 000106 1786064191.302408 trunc from 0x001fdb to 0x009403 (0x7429 bytes) 000107 1786064191.327715 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x33d38): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33d38): Mirror 2: 0xff0d187b (stale) Mirror 4: 0xff0d187b (stale) Mirror 5: 0x910486b6 (stale) CRC-32 checksum value for chunk [0, 0x33d38): Mirror 1: 0x4864d69f Mirror 6: 0x4864d69f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33d38: skipped 000108 1786064192.431011 write 0x028dc2 thru 0x02f78c (0x69cb bytes) 000109 1786064192.582666 write 0x01c217 thru 0x02745e (0xb248 bytes) 000110 1786064192.604211 trunc from 0x033d38 to 0x008b71 (0x2b1c6 bytes) 000111 1786064192.678873 write 0x01e432 thru 0x02754a (0x9119 bytes) 000112 1786064192.681286 punch from 0x012cf3 to 0x01c6f7 (0x9a05 bytes) 000113 1786064192.688482 mapwrite 0x02fe52 thru 0x0321d4 (0x2383 bytes) 000114 1786064192.764253 mapread 0x021323 thru 0x031143 (0xfe21 bytes) 000115 1786064192.772788 fallocate 0x017000 thru 0x01cd53 (0x5d54 bytes) 000116 1786064192.781992 trunc from 0x00ea20 to 0x01e149 (0xf72a bytes) 000118 1786064192.791416 trunc from 0x03099c to 0x0321d4 (0x1839 bytes) 000119 1786064192.797561 punch from 0x01ec58 to 0x02c9e2 (0xdd8b bytes) 000120 1786064192.803778 trunc from 0x0321d5 to 0x0009f8 (0x317dc bytes) 000121 1786064192.828872 read 0x00028e thru 0x0009f8 (0x76b bytes) 000122 1786064192.836105 punch from 0x000925 to 0x0009f8 (0xd4 bytes) 000123 1786064192.840834 mapwrite 0x0076c5 thru 0x00936b (0x1ca7 bytes) 000124 1786064192.893596 fallocate 0x01e85d thru 0x02c44f (0xdbf3 bytes) 000126 1786064192.901615 mapread 0x005cf6 thru 0x0126c6 (0xc9d1 bytes) 000127 1786064192.914907 fallocate 0x015abe thru 0x0205b9 (0xaafc bytes) truncating to largest ever: 0x3848e 000130 1786064192.923030 trunc from 0x02c450 to 0x03848d (0xc03e bytes) 000131 1786064192.950200 read 0x02f42d thru 0x036813 (0x73e7 bytes) 000132 1786064192.977710 mapwrite 0x033abc thru 0x03d6a8 (0x9bed bytes) 000133 1786064193.071274 trunc from 0x03d6a9 to 0x0251ca (0x184de bytes) 000134 1786064193.090806 mapread 0x01b00b thru 0x0251ca (0xa1c0 bytes) 000136 1786064193.093709 write 0x0111c3 thru 0x019aaa (0x88e8 bytes) 000138 1786064193.097516 fallocate 0x00b71d thru 0x00f17c (0x3a60 bytes) 000139 1786064193.108959 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x251cb): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x251cb): Mirror 2: 0x7f927157 (stale) Mirror 4: 0x7f927157 (stale) Mirror 5: 0xfdf51bf5 (stale) Mirror 6: 0x4fe311eb (stale) CRC-32 checksum value for chunk [0, 0x251cb): Mirror 1: 0x4ce4a2ef Mirror 7: 0x4ce4a2ef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x251cb: skipped 000140 1786064194.277720 mapwrite 0x035e21 thru 0x035e3c (0x1c bytes) 000141 1786064194.488577 mapwrite 0x00d1b4 thru 0x01bdad (0xebfa bytes) 000142 1786064194.579661 write 0x037dbb thru 0x03ffff (0x8245 bytes) 000143 1786064194.593722 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x52a1807a (stale) Mirror 4: 0x52a1807a (stale) Mirror 5: 0x272ff10b (stale) Mirror 6: 0xe82d4069 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x52a1807a (stale) Mirror 4: 0x52a1807a (stale) Mirror 5: 0x272ff10b (stale) Mirror 6: 0xe82d4069 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000144 1786064195.202579 trunc from 0x00bc29 to 0x0132e5 (0x76bd bytes) 000145 1786064195.210243 mapread 0x02d397 thru 0x039f50 (0xcbba bytes) 000146 1786064195.245292 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x52a1807a (stale) Mirror 5: 0x272ff10b (stale) Mirror 6: 0xe82d4069 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x52a1807a (stale) Mirror 5: 0x272ff10b (stale) Mirror 6: 0xe82d4069 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000147 1786064195.760775 trunc from 0x033aab to 0x0381f5 (0x474b bytes) 000149 1786064195.770760 trunc from 0x040000 to 0x019668 (0x26997 bytes) 000150 1786064195.809416 punch from 0x00e17f to 0x01541e (0x72a0 bytes) 000151 1786064195.822591 mapwrite 0x00ebd8 thru 0x01b1c0 (0xc5e9 bytes) 000152 1786064195.934293 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1b1c1): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b1c1): Mirror 2: 0x39fc2933 (stale) Mirror 5: 0xc1dc8170 (stale) Mirror 6: 0xcb99edde (stale) CRC-32 checksum value for chunk [0, 0x1b1c1): Mirror 1: 0x99d92f12 Mirror 7: 0x99d92f12 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b1c1: skipped 000153 1786064197.051177 mapwrite 0x00060d thru 0x002abe (0x24b2 bytes) 000154 1786064197.197540 mapread 0x0176d6 thru 0x01b1c0 (0x3aeb bytes) 000155 1786064197.208762 read 0x004fe2 thru 0x011332 (0xc351 bytes) 000156 1786064197.225247 MIRROR_RESYNC 5 mirrors 000157 1786064198.973840 mapread 0x002ff2 thru 0x00cafb (0x9b0a bytes) 000158 1786064198.997130 trunc from 0x01b1c1 to 0x031e78 (0x16cb8 bytes) 000159 1786064199.047103 trunc from 0x031e79 to 0x01dd4a (0x1412e bytes) 000160 1786064199.081227 trunc from 0x0156ee to 0x016ad4 (0x13e7 bytes) fallocating to largest ever: 0x40000 000161 1786064199.089649 fallocate 0x03ab79 thru 0x03ffff (0x5487 bytes) 000162 1786064199.096829 fallocate 0x034390 thru 0x03ffff (0xbc70 bytes) 000163 1786064199.104823 trunc from 0x02597a to 0x0308e2 (0xaf69 bytes) 000164 1786064199.113788 trunc from 0x040000 to 0x01dc4d (0x223b2 bytes) 000165 1786064199.139205 mapread 0x016692 thru 0x01784a (0x11b9 bytes) 000166 1786064199.154933 mapread 0x01d227 thru 0x01dc4d (0xa27 bytes) 000167 1786064199.156864 write 0x01790e thru 0x024a8e (0xd181 bytes) 000168 1786064199.162345 fallocate 0x02c332 thru 0x02ff60 (0x3c2f bytes) 000169 1786064199.169258 punch from 0x004e55 to 0x0112ef (0xc49b bytes) 000170 1786064199.176192 punch from 0x01a56c to 0x01b871 (0x1306 bytes) 000171 1786064199.206392 mapread 0x00ea0c thru 0x0132b2 (0x48a7 bytes) 000173 1786064199.224748 trunc from 0x02ff61 to 0x01bb75 (0x143eb bytes) 000174 1786064199.250876 fallocate 0x02a6d6 thru 0x031dc4 (0x76ef bytes) 000175 1786064199.259463 mapread 0x02aa54 thru 0x031dc4 (0x7371 bytes) 000176 1786064199.273223 trunc from 0x00876e to 0x0097af (0x1042 bytes) 000177 1786064199.284836 MIRROR_RESYNC 5 mirrors 000178 1786064200.629525 trunc from 0x01cbb0 to 0x029530 (0xc981 bytes) 000179 1786064200.714264 read 0x015165 thru 0x01acbb (0x5b57 bytes) 000180 1786064200.752183 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31dc5): Mirror 5: 0xd8ebb61f (stale) Mirror 6: 0xd8ebb61f (stale) Mirror 7: 0xd8ebb61f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31dc5): Mirror 5: 0xd8ebb61f (stale) Mirror 6: 0xd8ebb61f (stale) Mirror 7: 0xd8ebb61f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31dc5: skipped 000181 1786064201.382246 mapread 0x02a5a7 thru 0x031dc4 (0x781e bytes) 000182 1786064201.414439 mapread 0x00fe14 thru 0x01783d (0x7a2a bytes) 000183 1786064201.447886 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31dc5): Mirror 5: 0xd8ebb61f (stale) Mirror 6: 0xd8ebb61f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31dc5): Mirror 5: 0xd8ebb61f (stale) Mirror 6: 0xd8ebb61f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31dc5: skipped 000184 1786064201.924275 read 0x01ff41 thru 0x02fe72 (0xff32 bytes) 000185 1786064201.948732 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31dc5): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31dc5): Mirror 5: 0xd8ebb61f (stale) Mirror 6: 0xd8ebb61f (stale) CRC-32 checksum value for chunk [0, 0x31dc5): Mirror 1: 0xd8ebb61f Mirror 7: 0xd8ebb61f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31dc5: skipped 000186 1786064203.273125 mapwrite 0x009575 thru 0x01288c (0x9318 bytes) 000188 1786064203.421407 write 0x031490 thru 0x039b58 (0x86c9 bytes) 000189 1786064203.466556 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x39b59): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39b59): Mirror 5: 0x61c24567 (stale) Mirror 6: 0x61c24567 (stale) Mirror 7: 0x61c24567 (stale) CRC-32 checksum value for chunk [0, 0x39b59): Mirror 1: 0x69570a95 Mirror 8: 0x69570a95 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39b59: skipped 000190 1786064204.820232 write 0x011195 thru 0x01e686 (0xd4f2 bytes) 000191 1786064204.939417 punch from 0x01d369 to 0x01d473 (0x10b bytes) 000193 1786064204.979577 trunc from 0x039b59 to 0x0116ae (0x284aa bytes) 000194 1786064205.035671 fallocate 0x004700 thru 0x0142ac (0xfbad bytes) 000195 1786064205.053047 read 0x011f82 thru 0x0142ac (0x232b bytes) 000196 1786064205.067959 write 0x0113b3 thru 0x01ba39 (0xa687 bytes) 000197 1786064205.072973 write 0x021305 thru 0x022d41 (0x1a3d bytes) 000198 1786064205.076448 read 0x00ab65 thru 0x012600 (0x7a9c bytes) 000199 1786064205.090253 mapread 0x016a66 thru 0x01bccf (0x526a bytes) 000200 1786064205.092080 read 0x017b3e thru 0x01aada (0x2f9d bytes) 000201 1786064205.093453 punch from 0x0209c9 to 0x022d41 (0x2379 bytes) 000202 1786064205.116033 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22d42): Mirror 6: 0xae59b6f4 (stale) Mirror 7: 0xae59b6f4 (stale) Mirror 8: 0xe042a91b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22d42): Mirror 6: 0xae59b6f4 (stale) Mirror 7: 0xae59b6f4 (stale) Mirror 8: 0xe042a91b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22d42: skipped 000203 1786064205.592880 punch from 0x002a17 to 0x00477c (0x1d66 bytes) 000204 1786064205.599190 trunc from 0x01f7f4 to 0x022d41 (0x354e bytes) 000205 1786064205.610033 write 0x000f08 thru 0x00bf8f (0xb088 bytes) 000206 1786064205.641439 write 0x017925 thru 0x026f9f (0xf67b bytes) 000207 1786064205.655872 mapwrite 0x032c95 thru 0x03fc2b (0xcf97 bytes) 000208 1786064205.824576 trunc from 0x03fc2c to 0x01f533 (0x206f8 bytes) 000209 1786064205.860704 fallocate 0x03aac0 thru 0x03ffff (0x5540 bytes) 000210 1786064205.876327 fallocate 0x0368b7 thru 0x03ffff (0x9749 bytes) 000211 1786064205.886813 write 0x0019b1 thru 0x00fa2a (0xe07a bytes) 000212 1786064205.903800 read 0x00a98c thru 0x012391 (0x7a06 bytes) 000213 1786064205.919448 trunc from 0x016eba to 0x022d95 (0xbedc bytes) 000214 1786064205.935651 fallocate 0x02d305 thru 0x033d81 (0x6a7d bytes) 000215 1786064205.948872 trunc from 0x0051c4 to 0x007ac7 (0x2904 bytes) 000216 1786064205.969163 mapread 0x02f12d thru 0x037926 (0x87fa bytes) 000217 1786064205.990677 mapread 0x022904 thru 0x0246df (0x1ddc bytes) 000218 1786064206.004649 trunc from 0x040000 to 0x03d6f8 (0x2907 bytes) 000219 1786064206.025757 trunc from 0x03d6f9 to 0x032e15 (0xa8e3 bytes) 000220 1786064206.056667 mapread 0x021b0d thru 0x02ec9f (0xd193 bytes) 000221 1786064206.059410 write 0x036f4f thru 0x03c550 (0x5602 bytes) 000222 1786064206.069161 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3c551): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c551): Mirror 6: 0xa4adf152 (stale) Mirror 7: 0xa4adf152 (stale) Mirror 8: 0xb1e5d662 (stale) CRC-32 checksum value for chunk [0, 0x3c551): Mirror 1: 0xf90839d9 Mirror 9: 0xf90839d9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c551: skipped 000223 1786064207.408608 trunc from 0x03c551 to 0x00fe4f (0x2c701 bytes) 000224 1786064207.523933 fallocate 0x00d6fd thru 0x01c3f2 (0xecf6 bytes) 000225 1786064207.533184 punch from 0x005da7 to 0x00cb08 (0x6d62 bytes) 000227 1786064207.547080 mapread 0x005f3d thru 0x00fe4f (0x9f13 bytes) 000228 1786064207.612759 mapread 0x008dfd thru 0x00fe4f (0x7053 bytes) 000229 1786064207.615115 trunc from 0x00c600 to 0x00fe4f (0x3850 bytes) 000230 1786064207.622998 trunc from 0x00fe50 to 0x0073fb (0x8a54 bytes) 000231 1786064207.663958 punch from 0x000898 to 0x0073fb (0x6b64 bytes) 000232 1786064207.675382 trunc from 0x0073fc to 0x0100d8 (0x8cdd bytes) 000233 1786064207.721036 MIRROR_RESYNC 5 mirrors 000234 1786064208.904959 trunc from 0x00abd5 to 0x0100d8 (0x5504 bytes) 000235 1786064208.955050 read 0x0009f4 thru 0x00d811 (0xce1e bytes) 000236 1786064208.967511 read 0x00d343 thru 0x0100d8 (0x2d96 bytes) 000237 1786064208.981170 MIRROR_RESYNC 5 mirrors 000238 1786064210.088245 trunc from 0x00725d to 0x0100d8 (0x8e7c bytes) 000239 1786064210.130726 trunc from 0x004d2c to 0x007c9a (0x2f6f bytes) 000240 1786064210.138773 write 0x033f82 thru 0x03ffff (0xc07e bytes) 000241 1786064210.142145 punch from 0x01612f to 0x017053 (0xf25 bytes) 000242 1786064210.147599 mapwrite 0x01adf2 thru 0x01e13c (0x334b bytes) 000243 1786064210.192488 trunc from 0x040000 to 0x00e0b4 (0x31f4b bytes) 000244 1786064210.219330 trunc from 0x00bda6 to 0x00e0b4 (0x230f bytes) 000245 1786064210.225897 mapwrite 0x03ddce thru 0x03ef0a (0x113d bytes) 000246 1786064210.279965 read 0x015a97 thru 0x015f99 (0x503 bytes) 000247 1786064210.288154 trunc from 0x00ecd1 to 0x015e5d (0x718d bytes) 000248 1786064210.297114 read 0x0146af thru 0x022129 (0xda7b bytes) 000249 1786064210.311440 trunc from 0x03659a to 0x03dce1 (0x7748 bytes) 000250 1786064210.318012 fallocate 0x027041 thru 0x02921f (0x21df bytes) 000251 1786064210.325696 fallocate 0x0046e6 thru 0x007fbe (0x38d9 bytes) 000253 1786064210.334084 punch from 0x015ecd to 0x016144 (0x278 bytes) 000254 1786064210.343186 fallocate 0x021135 thru 0x02dcc0 (0xcb8c bytes) 000255 1786064210.356071 mapread 0x0107fd thru 0x01700b (0x680f bytes) 000256 1786064210.374107 trunc from 0x014e33 to 0x01b4f8 (0x66c6 bytes) 000257 1786064210.380681 punch from 0x0095f7 to 0x00fbce (0x65d8 bytes) 000258 1786064210.390461 read 0x0098e0 thru 0x00b391 (0x1ab2 bytes) 000259 1786064210.414582 write 0x02a30f thru 0x036682 (0xc374 bytes) 000260 1786064210.418025 trunc from 0x03ef0b to 0x03859a (0x6970 bytes) 000261 1786064210.462627 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3859b): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3859b): Mirror 6: 0x4794474b (stale) Mirror 7: 0x4794474b (stale) Mirror 8: 0x4794474b (stale) Mirror 9: 0x4794474b (stale) CRC-32 checksum value for chunk [0, 0x3859b): Mirror 1: 0x7184eda1 Mirror 10: 0x7184eda1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3859b: skipped 000262 1786064211.700131 fallocate 0x007517 thru 0x012803 (0xb2ed bytes) 000263 1786064211.826925 mapwrite 0x02734d thru 0x030e94 (0x9b48 bytes) 000264 1786064211.934638 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3859b): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3859b): Mirror 6: 0x4794474b (stale) Mirror 7: 0x4794474b (stale) Mirror 8: 0x4794474b (stale) Mirror 9: 0x4794474b (stale) Mirror 10: 0x7184eda1 (stale) CRC-32 checksum value for chunk [0, 0x3859b): Mirror 1: 0x4a32db9e Mirror 11: 0x4a32db9e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3859b: skipped 000266 1786064213.460825 read 0x031089 thru 0x03859a (0x7512 bytes) 000267 1786064213.489606 trunc from 0x02e5c9 to 0x031cd8 (0x3710 bytes) 000268 1786064213.550089 MIRROR_RESYNC 7 mirrors 000269 1786064215.340096 mapwrite 0x0176b1 thru 0x01e9a1 (0x72f1 bytes) 000270 1786064215.456228 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3859b): Mirror 6: 0x14dce118 (stale) Mirror 7: 0x14dce118 (stale) Mirror 8: 0x14dce118 (stale) Mirror 9: 0x14dce118 (stale) Mirror 10: 0x14dce118 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3859b): Mirror 6: 0x14dce118 (stale) Mirror 7: 0x14dce118 (stale) Mirror 8: 0x14dce118 (stale) Mirror 9: 0x14dce118 (stale) Mirror 10: 0x14dce118 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3859b: skipped 000271 1786064216.078216 MIRROR_RESYNC 6 mirrors 000272 1786064218.124804 punch from 0x007c38 to 0x009b7c (0x1f45 bytes) 000273 1786064218.176169 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3859b): Mirror 6: 0x72a6aa9d (stale) Mirror 7: 0x72a6aa9d (stale) Mirror 8: 0x72a6aa9d (stale) Mirror 10: 0x72a6aa9d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3859b): Mirror 6: 0x72a6aa9d (stale) Mirror 7: 0x72a6aa9d (stale) Mirror 8: 0x72a6aa9d (stale) Mirror 10: 0x72a6aa9d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3859b: skipped 000274 1786064218.825071 fallocate 0x02d9f5 thru 0x033eb4 (0x64c0 bytes) 000276 1786064218.845637 punch from 0x036724 to 0x03859a (0x1e77 bytes) 000277 1786064218.853858 trunc from 0x001579 to 0x00d588 (0xc010 bytes) 000278 1786064218.862096 write 0x031e23 thru 0x03531a (0x34f8 bytes) 000279 1786064218.898267 MIRROR_RESYNC 5 mirrors 000280 1786064220.803283 trunc from 0x03859b to 0x029b8e (0xea0c bytes) 000281 1786064220.858863 trunc from 0x029b8f to 0x0111f9 (0x18995 bytes) 000282 1786064220.896121 trunc from 0x0111fa to 0x03fa89 (0x2e890 bytes) 000283 1786064220.917578 mapwrite 0x027c83 thru 0x034582 (0xc900 bytes) 000284 1786064220.980847 punch from 0x00c96b to 0x01764b (0xace1 bytes) 000285 1786064220.990041 read 0x01d93c thru 0x028b38 (0xb1fd bytes) 000286 1786064221.002492 trunc from 0x03fa8a to 0x01c161 (0x23928 bytes) 000287 1786064221.023570 fallocate 0x02bb10 thru 0x037bff (0xc0f0 bytes) 000289 1786064221.031993 read 0x00e5e5 thru 0x01ac5e (0xc67a bytes) 000290 1786064221.078013 fallocate 0x01a7ba thru 0x02886b (0xe0b2 bytes) 000291 1786064221.084020 read 0x017f09 thru 0x0270ea (0xf1e2 bytes) 000292 1786064221.084909 mapwrite 0x01a9bd thru 0x01e2b7 (0x38fb bytes) 000293 1786064221.110155 trunc from 0x01973b to 0x0209b1 (0x7277 bytes) 000294 1786064221.117914 write 0x00aa30 thru 0x01a632 (0xfc03 bytes) 000295 1786064221.127056 punch from 0x034124 to 0x037bff (0x3adc bytes) 000296 1786064221.133475 mapread 0x02b48d thru 0x02edd5 (0x3949 bytes) 000297 1786064221.135505 mapwrite 0x03f3fa thru 0x03ffff (0xc06 bytes) 000298 1786064221.197046 read 0x006336 thru 0x0158dd (0xf5a8 bytes) 000299 1786064221.204029 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x179219c5 (stale) Mirror 8: 0x179219c5 (stale) Mirror 10: 0x179219c5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x179219c5 (stale) Mirror 8: 0x179219c5 (stale) Mirror 10: 0x179219c5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000300 1786064221.638774 fallocate 0x0142d7 thru 0x01a236 (0x5f60 bytes) 000301 1786064221.644704 write 0x029591 thru 0x0376e8 (0xe158 bytes) 000302 1786064221.667476 mapwrite 0x00d2c3 thru 0x01cf25 (0xfc63 bytes) 000303 1786064221.728134 mapwrite 0x01bf2e thru 0x0292a1 (0xd374 bytes) 000305 1786064221.766418 MIRROR_RESYNC 4 mirrors 000307 1786064223.062458 punch from 0x00f7f3 to 0x014296 (0x4aa4 bytes) 000308 1786064223.114032 fallocate 0x003a84 thru 0x00604c (0x25c9 bytes) 000309 1786064223.121101 read 0x00e71e thru 0x00ed67 (0x64a bytes) 000310 1786064223.130006 write 0x0221d8 thru 0x028536 (0x635f bytes) 000311 1786064223.140962 trunc from 0x040000 to 0x023e60 (0x1c19f bytes) 000313 1786064223.177764 trunc from 0x023e61 to 0x016cab (0xd1b5 bytes) 000314 1786064223.193456 mapread 0x01022b thru 0x0145e9 (0x43bf bytes) 000315 1786064223.222599 fallocate 0x0326fe thru 0x03ffff (0xd902 bytes) 000316 1786064223.232130 write 0x01b44d thru 0x01b6ce (0x282 bytes) 000318 1786064223.240482 trunc from 0x00c658 to 0x01c545 (0xfeee bytes) 000319 1786064223.259378 punch from 0x0143fd to 0x014ee3 (0xae7 bytes) 000321 1786064223.270015 read 0x037f23 thru 0x03d9f0 (0x5ace bytes) 000322 1786064223.289593 fallocate 0x003aa3 thru 0x010e36 (0xd394 bytes) 000323 1786064223.295801 mapwrite 0x00cf51 thru 0x014a2f (0x7adf bytes) 000324 1786064223.327601 trunc from 0x040000 to 0x0133ce (0x2cc31 bytes) 000325 1786064223.365975 write 0x000bdd thru 0x009b5b (0x8f7f bytes) 000326 1786064223.381817 MIRROR_RESYNC 4 mirrors 000327 1786064224.276952 mapread 0x006342 thru 0x00c6c7 (0x6386 bytes) 000328 1786064224.301552 fallocate 0x01dfaa thru 0x02bb5a (0xdbb1 bytes) 000329 1786064224.351163 mapread 0x0083be thru 0x009b0d (0x1750 bytes) 000330 1786064224.367753 fallocate 0x01ffae thru 0x02271c (0x276f bytes) 000332 1786064224.374199 read 0x00f31e thru 0x00f9be (0x6a1 bytes) 000333 1786064224.375063 read 0x003047 thru 0x0122af (0xf269 bytes) 000334 1786064224.384984 trunc from 0x0133cf to 0x01b245 (0x7e77 bytes) 000335 1786064224.458946 read 0x00498a thru 0x009989 (0x5000 bytes) 000336 1786064224.463852 punch from 0x00c168 to 0x01487d (0x8716 bytes) 000337 1786064224.474477 trunc from 0x00f3c4 to 0x01b0ec (0xbd29 bytes) 000339 1786064224.483513 read 0x01941c thru 0x01b245 (0x1e2a bytes) 000340 1786064224.497584 punch from 0x010349 to 0x0152a5 (0x4f5d bytes) 000342 1786064224.504855 read 0x000c5f thru 0x00e235 (0xd5d7 bytes) 000343 1786064224.505598 mapwrite 0x031c9f thru 0x03ffff (0xe361 bytes) 000344 1786064224.605809 write 0x0295d0 thru 0x02fc92 (0x66c3 bytes) 000345 1786064224.609638 trunc from 0x02cb15 to 0x03a36c (0xd858 bytes) 000346 1786064224.634913 write 0x014829 thru 0x02480e (0xffe6 bytes) 000347 1786064224.638706 read 0x002b2d thru 0x0100e6 (0xd5ba bytes) 000348 1786064224.640156 mapread 0x0281a2 thru 0x02b0c6 (0x2f25 bytes) 000349 1786064224.641783 trunc from 0x040000 to 0x019957 (0x266a8 bytes) 000350 1786064224.669492 mapread 0x0119db thru 0x01364d (0x1c73 bytes) 000351 1786064224.671348 write 0x02c5bf thru 0x031c6d (0x56af bytes) 000352 1786064224.673375 write 0x015865 thru 0x022c34 (0xd3d0 bytes) 000353 1786064224.688211 MIRROR_RESYNC 4 mirrors 000354 1786064226.054284 mapread 0x010f7e thru 0x018bbb (0x7c3e bytes) 000357 1786064226.085281 fallocate 0x0340ac thru 0x03e1e5 (0xa13a bytes) 000358 1786064226.148144 write 0x0109ad thru 0x0169ea (0x603e bytes) 000359 1786064226.163256 trunc from 0x03e1e6 to 0x016df5 (0x273f0 bytes) 000360 1786064226.198786 trunc from 0x016df6 to 0x033f74 (0x1d17f bytes) 000361 1786064226.228199 trunc from 0x02077c to 0x023946 (0x31cb bytes) 000362 1786064226.236448 mapwrite 0x01108f thru 0x0146c8 (0x363a bytes) 000363 1786064226.264927 trunc from 0x033f75 to 0x02c8af (0x76c5 bytes) 000364 1786064226.302139 MIRROR_RESYNC 4 mirrors 000365 1786064227.805869 fallocate 0x00acd3 thru 0x011a27 (0x6d55 bytes) 000366 1786064227.856266 mapwrite 0x0099cb thru 0x013bfe (0xa234 bytes) 000367 1786064227.907038 MIRROR_RESYNC 4 mirrors 000369 1786064228.812850 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) 000370 1786064229.238864 trunc from 0x006e88 to 0x011518 (0xa691 bytes) 000371 1786064229.299859 mapwrite 0x008859 thru 0x01250a (0x9cb2 bytes) 000372 1786064229.352166 punch from 0x017338 to 0x01844e (0x1117 bytes) 000373 1786064229.360283 trunc from 0x004dcd to 0x01260e (0xd842 bytes) 000374 1786064229.370152 fallocate 0x0216f1 thru 0x030cc2 (0xf5d2 bytes) 000376 1786064229.383789 MIRROR_RESYNC 3 mirrors 000377 1786064230.184248 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) 000378 1786064230.629936 trunc from 0x02c8b0 to 0x00cec3 (0x1f9ec bytes) 000379 1786064230.753638 punch from 0x003bf2 to 0x0083b5 (0x47c4 bytes) 000380 1786064230.763925 write 0x034d60 thru 0x03ffff (0xb2a0 bytes) 000381 1786064230.769823 mapread 0x030608 thru 0x03fad0 (0xf4c9 bytes) 000382 1786064230.805832 punch from 0x01f6b1 to 0x022072 (0x29c2 bytes) 000383 1786064230.814769 trunc from 0x025e5e to 0x02acf8 (0x4e9b bytes) 000384 1786064230.829881 fallocate 0x0193c6 thru 0x020e38 (0x7a73 bytes) 000385 1786064230.842792 mapread 0x02333e thru 0x0246bb (0x137e bytes) 000387 1786064230.889956 punch from 0x029180 to 0x02f519 (0x639a bytes) 000388 1786064230.904976 mapwrite 0x03fb87 thru 0x03ffff (0x479 bytes) 000389 1786064230.960262 write 0x012b06 thru 0x017c77 (0x5172 bytes) 000390 1786064230.987903 write 0x02b3f5 thru 0x03a80a (0xf416 bytes) 000391 1786064231.005448 trunc from 0x021b9c to 0x026da0 (0x5205 bytes) 000392 1786064231.018987 mapwrite 0x015f8e thru 0x01bddd (0x5e50 bytes) 000393 1786064231.081916 read 0x014651 thru 0x014a49 (0x3f9 bytes) 000394 1786064231.083482 trunc from 0x040000 to 0x018b78 (0x27487 bytes) 000395 1786064231.114894 mapwrite 0x0390f4 thru 0x03d7f6 (0x4703 bytes) 000396 1786064231.177832 mapread 0x02aa51 thru 0x0356ad (0xac5d bytes) 000397 1786064231.200906 write 0x03609b thru 0x03cfe5 (0x6f4b bytes) 000398 1786064231.203342 fallocate 0x008de2 thru 0x00b2bd (0x24dc bytes) 000399 1786064231.209739 mapwrite 0x01b496 thru 0x01b65f (0x1ca bytes) 000400 1786064231.250185 trunc from 0x03d7f7 to 0x035780 (0x8076 bytes) 000401 1786064231.280780 trunc from 0x01321a to 0x02301b (0xfe02 bytes) 000402 1786064231.293932 mapread 0x001770 thru 0x0116ef (0xff80 bytes) 000403 1786064231.323130 trunc from 0x035781 to 0x00eb1f (0x26c61 bytes) 000405 1786064231.371864 MIRROR_RESYNC 2 mirrors 000406 1786064232.181138 write 0x038f19 thru 0x03fd96 (0x6e7e bytes) 000407 1786064232.232937 punch from 0x033478 to 0x03fa86 (0xc60f bytes) 000408 1786064232.259199 fallocate 0x02d9a8 thru 0x033af4 (0x614d bytes) 000410 1786064232.267771 punch from 0x0231d1 to 0x030f2c (0xdd5c bytes) 000411 1786064232.275219 punch from 0x009cc8 to 0x00b64c (0x1985 bytes) 000413 1786064232.285188 mapwrite 0x036f36 thru 0x03ffff (0x90ca bytes) 000414 1786064232.396632 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) 000415 1786064232.608206 read 0x02b51e thru 0x03ac05 (0xf6e8 bytes) 000416 1786064232.640713 trunc from 0x040000 to 0x02897e (0x17681 bytes) 000417 1786064232.678906 punch from 0x003272 to 0x004e46 (0x1bd5 bytes) 000418 1786064232.696803 MIRROR_EXTEND to 2 mirrors 000419 1786064233.693273 fallocate 0x002f44 thru 0x00b7ff (0x88bc bytes) 000420 1786064233.748064 mapread 0x013ec8 thru 0x0207ad (0xc8e6 bytes) 000421 1786064233.796114 mapread 0x024d67 thru 0x02897e (0x3c18 bytes) 000422 1786064233.804471 trunc from 0x02897f to 0x01f31e (0x9660 bytes) 000423 1786064233.844251 mapread 0x007bc1 thru 0x009825 (0x1c65 bytes) 000424 1786064233.869449 MIRROR_RESYNC 2 mirrors 000425 1786064234.530910 write 0x038031 thru 0x03ffff (0x7fcf bytes) 000426 1786064234.568667 write 0x00297e thru 0x002ab4 (0x137 bytes) 000428 1786064234.576581 fallocate 0x021496 thru 0x029831 (0x839c bytes) 000429 1786064234.582594 trunc from 0x040000 to 0x01831a (0x27ce5 bytes) 000430 1786064234.618587 mapwrite 0x03b265 thru 0x03d3b6 (0x2152 bytes) 000431 1786064234.756228 write 0x029e01 thru 0x02ecfa (0x4efa bytes) 000432 1786064234.785281 punch from 0x036c8b to 0x03c852 (0x5bc8 bytes) 000433 1786064234.797562 mapwrite 0x00e2b6 thru 0x00e67a (0x3c5 bytes) 000434 1786064234.872529 write 0x02ee5a thru 0x0357a7 (0x694e bytes) 000435 1786064234.903727 read 0x02829f thru 0x0306fb (0x845d bytes) 000436 1786064234.916355 write 0x021a26 thru 0x028176 (0x6751 bytes) 000437 1786064234.938828 write 0x01a794 thru 0x028c4b (0xe4b8 bytes) 000438 1786064234.958683 trunc from 0x03d3b7 to 0x02208c (0x1b32a bytes) 000439 1786064235.020677 MIRROR_RESYNC 2 mirrors 000440 1786064235.920260 read 0x00a20f thru 0x017fec (0xddde bytes) 000443 1786064235.942140 punch from 0x01292b to 0x013d80 (0x1456 bytes) 000444 1786064235.991367 punch from 0x0064fd to 0x01354f (0xd053 bytes) 000445 1786064235.999617 punch from 0x01607f to 0x02208c (0xc00e bytes) 000447 1786064236.012501 mapwrite 0x00c327 thru 0x0179ef (0xb6c9 bytes) 000448 1786064236.076202 trunc from 0x015686 to 0x01c267 (0x6be2 bytes) 000449 1786064236.081972 mapwrite 0x017230 thru 0x01fddd (0x8bae bytes) 000450 1786064236.116569 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) 000451 1786064236.435032 punch from 0x00ba3d to 0x0108e7 (0x4eab bytes) 000452 1786064236.445886 read 0x011ce7 thru 0x014041 (0x235b bytes) 000453 1786064236.472427 MIRROR_EXTEND to 2 mirrors 000454 1786064237.935488 trunc from 0x0093fc to 0x00e3e5 (0x4fea bytes) 000455 1786064237.990938 fallocate 0x01b00d thru 0x024a44 (0x9a38 bytes) 000456 1786064237.997772 trunc from 0x01a438 to 0x01c8b0 (0x2479 bytes) 000457 1786064238.007219 trunc from 0x02208d to 0x032c09 (0x10b7d bytes) 000459 1786064238.058687 write 0x005f2b thru 0x0115cc (0xb6a2 bytes) 000460 1786064238.081345 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x32c0a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32c0a): Mirror 2: 0xfcd0b5bc (stale) CRC-32 checksum value for chunk [0, 0x32c0a): Mirror 1: 0xbc945cb0 Mirror 3: 0xbc945cb0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32c0a: skipped 000462 1786064239.401958 write 0x01a666 thru 0x027330 (0xcccb bytes) 000463 1786064239.467623 read 0x00d34d thru 0x00fe7f (0x2b33 bytes) 000464 1786064239.477909 write 0x033367 thru 0x03cb12 (0x97ac bytes) 000465 1786064239.479823 fallocate 0x020a0a thru 0x028cc2 (0x82b9 bytes) 000466 1786064239.486699 mapwrite 0x00366b thru 0x00ee33 (0xb7c9 bytes) 000467 1786064239.549719 fallocate 0x01a58b thru 0x023335 (0x8dab bytes) 000468 1786064239.564785 trunc from 0x03cb13 to 0x034c48 (0x7eca bytes) 000469 1786064239.602229 write 0x027e39 thru 0x035006 (0xd1ce bytes) 000470 1786064239.606950 trunc from 0x035007 to 0x029ce7 (0xb31f bytes) 000471 1786064239.644156 punch from 0x01fc7f to 0x029ce7 (0xa069 bytes) 000472 1786064239.672663 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29ce8): Mirror 2: 0x98e0ca7b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29ce8): Mirror 2: 0x98e0ca7b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29ce8: skipped 000473 1786064240.085530 trunc from 0x029ce8 to 0x028d7d (0xf6a bytes) 000474 1786064240.117653 write 0x005226 thru 0x008b9f (0x397a bytes) 000475 1786064240.130623 punch from 0x018352 to 0x01dddd (0x5a8c bytes) 000476 1786064240.139657 mapread 0x011443 thru 0x012e68 (0x1a26 bytes) 000477 1786064240.151735 mapread 0x0263c7 thru 0x028d7d (0x29b7 bytes) 000478 1786064240.167432 trunc from 0x01d26e to 0x0257d0 (0x8563 bytes) 000480 1786064240.179723 MIRROR_RESYNC 2 mirrors 000481 1786064240.917277 trunc from 0x028d7e to 0x019c16 (0xf167 bytes) 000482 1786064241.044812 punch from 0x00026a to 0x00aa28 (0xa7bf bytes) 000483 1786064241.057215 mapread 0x00ed72 thru 0x017aa2 (0x8d31 bytes) 000484 1786064241.085552 mapwrite 0x03707c thru 0x037e41 (0xdc6 bytes) 000485 1786064241.152289 write 0x0047ea thru 0x00d337 (0x8b4e bytes) 000486 1786064241.165686 write 0x02d83d thru 0x02ed04 (0x14c8 bytes) 000487 1786064241.171549 fallocate 0x01fa2e thru 0x024544 (0x4b17 bytes) 000488 1786064241.181706 mapread 0x0077f2 thru 0x00990a (0x2119 bytes) 000489 1786064241.182980 fallocate 0x02f628 thru 0x03915e (0x9b37 bytes) 000490 1786064241.191133 fallocate 0x0110ee thru 0x012011 (0xf24 bytes) 000491 1786064241.197524 read 0x01ae57 thru 0x01f0ff (0x42a9 bytes) 000493 1786064241.210112 read 0x01c350 thru 0x025743 (0x93f4 bytes) 000494 1786064241.223725 write 0x008911 thru 0x00f819 (0x6f09 bytes) 000495 1786064241.226935 read 0x02d5af thru 0x0333ef (0x5e41 bytes) 000496 1786064241.228489 write 0x0098aa thru 0x016cb1 (0xd408 bytes) 000497 1786064241.231434 read 0x032026 thru 0x037e41 (0x5e1c bytes) 000498 1786064241.233019 write 0x00b50a thru 0x014f5c (0x9a53 bytes) 000499 1786064241.236565 mapwrite 0x03ebf0 thru 0x03ffff (0x1410 bytes) 000500 1786064241.347558 fallocate 0x017667 thru 0x018403 (0xd9d bytes) 000501 1786064241.362496 read 0x0040f7 thru 0x0050ca (0xfd4 bytes) 000503 1786064241.379141 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) 000504 1786064241.634636 punch from 0x00050b to 0x00e17f (0xdc75 bytes) 000505 1786064241.646141 punch from 0x00421b to 0x0057fd (0x15e3 bytes) 000506 1786064241.658013 mapread 0x0216c4 thru 0x0299d5 (0x8312 bytes) 000507 1786064241.694013 read 0x002b95 thru 0x00b92a (0x8d96 bytes) 000508 1786064241.707923 write 0x0145cb thru 0x015655 (0x108b bytes) 000509 1786064241.739192 trunc from 0x02588f to 0x0305ac (0xad1e bytes) 000510 1786064241.753712 write 0x03afe2 thru 0x03ffff (0x501e bytes) 000511 1786064241.776891 write 0x000014 thru 0x00cbc5 (0xcbb2 bytes) 000512 1786064241.806130 write 0x013f1d thru 0x01805c (0x4140 bytes) 000513 1786064241.843279 MIRROR_EXTEND to 2 mirrors 000514 1786064243.359977 mapwrite 0x02a5f9 thru 0x02f652 (0x505a bytes) 000515 1786064243.502019 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) 000516 1786064243.744672 mapwrite 0x02ff31 thru 0x03b5c5 (0xb695 bytes) 000517 1786064243.861604 trunc from 0x040000 to 0x023320 (0x1ccdf bytes) 000519 1786064243.923056 write 0x02e0e1 thru 0x0326e4 (0x4604 bytes) 000520 1786064243.926364 trunc from 0x0326e5 to 0x0308ca (0x1e1a bytes) 000521 1786064243.963597 read 0x01c956 thru 0x02304f (0x66fa bytes) 000523 1786064243.990835 mapwrite 0x0287f8 thru 0x03862a (0xfe33 bytes) 000524 1786064244.129059 write 0x011bbe thru 0x0163ae (0x47f1 bytes) 000525 1786064244.155761 fallocate 0x00bdc9 thru 0x01167a (0x58b2 bytes) 000526 1786064244.176334 fallocate 0x01418c thru 0x02191e (0xd793 bytes) 000528 1786064244.183919 mapwrite 0x00e116 thru 0x00f419 (0x1304 bytes) 000529 1786064244.237623 fallocate 0x00f40b thru 0x013ab9 (0x46af bytes) 000531 1786064244.258501 punch from 0x026941 to 0x02dbd9 (0x7299 bytes) 000532 1786064244.269516 trunc from 0x0017eb to 0x00ab97 (0x93ad bytes) 000533 1786064244.281745 trunc from 0x03862b to 0x0191ca (0x1f460 bytes) 000535 1786064244.315761 fallocate 0x030b74 thru 0x03630a (0x5797 bytes) 000536 1786064244.334291 MIRROR_EXTEND to 2 mirrors 000537 1786064245.220784 fallocate 0x026d7e thru 0x02ab17 (0x3d9a bytes) 000538 1786064245.268283 read 0x029d13 thru 0x03630a (0xc5f8 bytes) 000539 1786064245.294659 trunc from 0x03630b to 0x00fcd8 (0x26632 bytes) 000540 1786064245.341050 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xfcd9): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfcd9): Mirror 2: 0x583b5a67 (stale) CRC-32 checksum value for chunk [0, 0xfcd9): Mirror 1: 0x583b5a67 Mirror 3: 0x583b5a67 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfcd9: skipped 000541 1786064246.427513 read 0x00e6d0 thru 0x00fcd8 (0x1609 bytes) 000542 1786064246.445855 mapwrite 0x01677c thru 0x016d70 (0x5f5 bytes) 000543 1786064246.596002 write 0x024afa thru 0x02a93e (0x5e45 bytes) 000544 1786064246.598561 mapwrite 0x0287fc thru 0x030203 (0x7a08 bytes) 000545 1786064246.674937 mapwrite 0x03d9cd thru 0x03ffff (0x2633 bytes) 000546 1786064246.764813 fallocate 0x02e6da thru 0x033f92 (0x58b9 bytes) 000547 1786064246.772546 read 0x02bb1b thru 0x02d1f1 (0x16d7 bytes) 000548 1786064246.773243 trunc from 0x040000 to 0x035aa4 (0xa55b bytes) 000549 1786064246.814226 mapwrite 0x01450d thru 0x0212e1 (0xcdd5 bytes) 000550 1786064246.855236 trunc from 0x035aa5 to 0x01f799 (0x1630b bytes) 000551 1786064246.887331 punch from 0x003272 to 0x00c721 (0x94b0 bytes) 000554 1786064246.896108 mapread 0x0109c4 thru 0x017fbb (0x75f8 bytes) 000555 1786064246.899566 trunc from 0x00f99e to 0x01ccc8 (0xd32b bytes) 000556 1786064246.906555 write 0x02d01f thru 0x031b00 (0x4ae2 bytes) 000557 1786064246.908500 mapread 0x0141b2 thru 0x01b7c0 (0x760f bytes) 000558 1786064246.956449 mapread 0x0132b6 thru 0x020210 (0xcf5b bytes) 000559 1786064246.974430 mapwrite 0x02462c thru 0x02df5b (0x9930 bytes) 000560 1786064247.055991 MIRROR_RESYNC 3 mirrors 000561 1786064248.238563 fallocate 0x02483b thru 0x02bf3a (0x7700 bytes) 000562 1786064248.278368 mapread 0x00fc18 thru 0x016730 (0x6b19 bytes) 000564 1786064248.306608 write 0x0110ac thru 0x01ba78 (0xa9cd bytes) 000565 1786064248.313567 trunc from 0x031b01 to 0x004824 (0x2d2dc bytes) 000566 1786064248.338927 write 0x00ca50 thru 0x01845f (0xba10 bytes) 000567 1786064248.342038 fallocate 0x01892d thru 0x01c2f4 (0x39c8 bytes) 000568 1786064248.349746 mapwrite 0x024b3c thru 0x033782 (0xec47 bytes) 000569 1786064248.434159 read 0x0112b9 thru 0x017746 (0x648e bytes) 000570 1786064248.440741 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33783): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33783): Mirror 2: 0x5b42ff1e (stale) Mirror 3: 0x5b42ff1e (stale) CRC-32 checksum value for chunk [0, 0x33783): Mirror 1: 0xb6f90ee0 Mirror 4: 0xb6f90ee0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33783: skipped 000571 1786064249.328845 trunc from 0x033783 to 0x009055 (0x2a72d bytes) 000572 1786064249.442119 write 0x0058ad thru 0x00b3dc (0x5b30 bytes) 000573 1786064249.451284 trunc from 0x003eeb to 0x005ad2 (0x1be8 bytes) 000574 1786064249.488801 MIRROR_RESYNC 4 mirrors 000575 1786064250.808510 trunc from 0x00b3dd to 0x0254ec (0x1a110 bytes) 000576 1786064250.872836 trunc from 0x00a0bc to 0x00a635 (0x57a bytes) 000577 1786064250.879933 trunc from 0x01071e to 0x01f40f (0xecf2 bytes) 000578 1786064250.885807 read 0x004ec2 thru 0x00701d (0x215c bytes) 000579 1786064250.893162 trunc from 0x00be48 to 0x01624a (0xa403 bytes) 000580 1786064250.901493 punch from 0x00fce5 to 0x015609 (0x5925 bytes) 000581 1786064250.907163 punch from 0x01fbe3 to 0x022804 (0x2c22 bytes) 000582 1786064250.915906 fallocate 0x002b3e thru 0x00e0a1 (0xb564 bytes) 000583 1786064250.921199 read 0x01fdf5 thru 0x0254ec (0x56f8 bytes) 000584 1786064250.946162 trunc from 0x0254ed to 0x029c58 (0x476c bytes) 000585 1786064250.975557 write 0x029b8b thru 0x031f1d (0x8393 bytes) 000586 1786064250.984933 mapread 0x00424f thru 0x008051 (0x3e03 bytes) 000587 1786064250.986518 write 0x02bd58 thru 0x0348ef (0x8b98 bytes) 000588 1786064250.988788 read 0x034774 thru 0x0348ef (0x17c bytes) 000589 1786064250.989734 read 0x014a4a thru 0x01f28d (0xa844 bytes) 000590 1786064250.990675 punch from 0x03206e to 0x0348ef (0x2882 bytes) 000591 1786064251.013272 trunc from 0x0348f0 to 0x019baf (0x1ad40 bytes) 000593 1786064251.042676 mapread 0x0158df thru 0x019baf (0x42d1 bytes) 000594 1786064251.044897 write 0x006caf thru 0x00c9a4 (0x5cf6 bytes) 000595 1786064251.046811 read 0x004316 thru 0x012aed (0xe7d8 bytes) 000596 1786064251.048961 fallocate 0x0295a5 thru 0x029671 (0xcd bytes) 000597 1786064251.056087 mapread 0x0164be thru 0x0228f6 (0xc439 bytes) 000598 1786064251.075157 read 0x0036b5 thru 0x007362 (0x3cae bytes) 000599 1786064251.076806 mapread 0x01028c thru 0x01e92b (0xe6a0 bytes) 000600 1786064251.082097 trunc from 0x029672 to 0x012745 (0x16f2c bytes) 000601 1786064251.127971 read 0x0046b1 thru 0x00537e (0xcce bytes) 000602 1786064251.135120 mapwrite 0x024acb thru 0x0300b4 (0xb5ea bytes) 000603 1786064251.246295 mapread 0x00d4c8 thru 0x015c62 (0x879b bytes) 000604 1786064251.248937 read 0x0089f6 thru 0x00cd70 (0x437b bytes) 000605 1786064251.251141 punch from 0x0046f7 to 0x00e3d7 (0x9ce1 bytes) 000606 1786064251.278968 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x300b5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x300b5): Mirror 2: 0xe6bcb643 (stale) Mirror 3: 0xe6bcb643 (stale) Mirror 4: 0xe6bcb643 (stale) CRC-32 checksum value for chunk [0, 0x300b5): Mirror 1: 0x91143b38 Mirror 5: 0x91143b38 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x300b5: skipped 000607 1786064252.470109 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x300b5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x300b5): Mirror 2: 0xe6bcb643 (stale) Mirror 3: 0xe6bcb643 (stale) Mirror 4: 0xe6bcb643 (stale) CRC-32 checksum value for chunk [0, 0x300b5): Mirror 1: 0x91143b38 Mirror 5: 0x91143b38 Mirror 6: 0x91143b38 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x300b5: skipped 000608 1786064253.866111 mapread 0x024932 thru 0x02fbc2 (0xb291 bytes) 000609 1786064253.902078 MIRROR_RESYNC 6 mirrors 000610 1786064255.331020 trunc from 0x0300b5 to 0x022c04 (0xd4b0 bytes) 000611 1786064255.454083 MIRROR_RESYNC 6 mirrors 000612 1786064257.270068 punch from 0x020724 to 0x022c04 (0x24e1 bytes) 000613 1786064257.318620 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22c05): Mirror 2: 0x97c29503 (stale) Mirror 3: 0x97c29503 (stale) Mirror 5: 0x97c29503 (stale) Mirror 6: 0x97c29503 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22c05): Mirror 2: 0x97c29503 (stale) Mirror 3: 0x97c29503 (stale) Mirror 5: 0x97c29503 (stale) Mirror 6: 0x97c29503 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22c05: skipped 000615 1786064258.000257 read 0x0123d8 thru 0x01d434 (0xb05d bytes) 000617 1786064258.013224 trunc from 0x022c05 to 0x02ac6f (0x806b bytes) 000618 1786064258.048966 mapread 0x01d165 thru 0x024640 (0x74dc bytes) 000619 1786064258.075473 mapread 0x010556 thru 0x0154b5 (0x4f60 bytes) 000620 1786064258.089821 fallocate 0x036167 thru 0x03dcc3 (0x7b5d bytes) 000621 1786064258.096248 mapread 0x03d27d thru 0x03dcc3 (0xa47 bytes) 000623 1786064258.109249 trunc from 0x03dcc4 to 0x03553a (0x8789 bytes) 000624 1786064258.127196 fallocate 0x03fcbf thru 0x03ffff (0x341 bytes) 000625 1786064258.133212 fallocate 0x02760b thru 0x02cb13 (0x5509 bytes) 000626 1786064258.142009 write 0x024931 thru 0x02db64 (0x9234 bytes) 000628 1786064258.151158 trunc from 0x040000 to 0x0366f4 (0x990b bytes) 000629 1786064258.173212 punch from 0x035222 to 0x0366f4 (0x14d3 bytes) 000630 1786064258.179731 trunc from 0x014e18 to 0x022c56 (0xde3f bytes) 000631 1786064258.189832 MIRROR_RESYNC 5 mirrors 000632 1786064260.033586 read 0x035c6f thru 0x0366f4 (0xa86 bytes) 000633 1786064260.040959 mapread 0x0158da thru 0x01d69b (0x7dc2 bytes) 000634 1786064260.060097 read 0x034715 thru 0x0366f4 (0x1fe0 bytes) 000636 1786064260.095082 MIRROR_EXTEND to 6 mirrors 000637 1786064261.334344 mapwrite 0x001b97 thru 0x00763c (0x5aa6 bytes) 000638 1786064261.478180 write 0x038b83 thru 0x03ffff (0x747d bytes) 000639 1786064261.491233 punch from 0x024e60 to 0x03098f (0xbb30 bytes) 000640 1786064261.508044 trunc from 0x040000 to 0x01ee0b (0x211f4 bytes) 000641 1786064261.553141 mapread 0x002a74 thru 0x00b93f (0x8ecc bytes) 000642 1786064261.557635 write 0x0399ca thru 0x03ffff (0x6636 bytes) 000643 1786064261.560702 mapwrite 0x006ff1 thru 0x014e46 (0xde56 bytes) 000644 1786064261.604348 write 0x02a300 thru 0x02abc5 (0x8c6 bytes) 000645 1786064261.610793 trunc from 0x040000 to 0x03bb18 (0x44e7 bytes) 000646 1786064261.633416 fallocate 0x016672 thru 0x017242 (0xbd1 bytes) 000647 1786064261.640107 write 0x03934d thru 0x03ffff (0x6cb3 bytes) 000648 1786064261.641952 write 0x0337bc thru 0x038077 (0x48bc bytes) 000649 1786064261.653878 trunc from 0x040000 to 0x02e559 (0x11aa6 bytes) 000650 1786064261.673133 mapwrite 0x004930 thru 0x011742 (0xce13 bytes) 000651 1786064261.713697 MIRROR_RESYNC 6 mirrors 000652 1786064263.181956 write 0x02f22b thru 0x036dec (0x7bc2 bytes) 000653 1786064263.215689 write 0x0104e2 thru 0x01ce4e (0xc96d bytes) 000654 1786064263.232175 mapread 0x02e39b thru 0x036dec (0x8a52 bytes) 000655 1786064263.254183 mapread 0x0173b5 thru 0x01e963 (0x75af bytes) 000656 1786064263.279247 fallocate 0x00e166 thru 0x017616 (0x94b1 bytes) 000657 1786064263.285086 read 0x024513 thru 0x030935 (0xc423 bytes) 000658 1786064263.294989 fallocate 0x000f1c thru 0x00ff93 (0xf078 bytes) 000659 1786064263.306193 fallocate 0x02ebab thru 0x03c38d (0xd7e3 bytes) 000660 1786064263.316064 mapwrite 0x025f56 thru 0x02f7dc (0x9887 bytes) 000661 1786064263.348618 trunc from 0x015744 to 0x01a759 (0x5016 bytes) 000662 1786064263.357690 mapwrite 0x02ee44 thru 0x03050e (0x16cb bytes) 000663 1786064263.377835 fallocate 0x009131 thru 0x0121f0 (0x90c0 bytes) 000664 1786064263.389237 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36ded): Mirror 2: 0xc116f06d (stale) Mirror 3: 0xc116f06d (stale) Mirror 5: 0xc116f06d (stale) Mirror 7: 0xc116f06d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ded): Mirror 2: 0xc116f06d (stale) Mirror 3: 0xc116f06d (stale) Mirror 5: 0xc116f06d (stale) Mirror 7: 0xc116f06d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ded: skipped 000665 1786064263.806775 fallocate 0x02fb99 thru 0x03748f (0x78f7 bytes) 000666 1786064263.818264 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x36ded): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ded): Mirror 2: 0xc116f06d (stale) Mirror 3: 0xc116f06d (stale) Mirror 5: 0xc116f06d (stale) Mirror 7: 0xc116f06d (stale) CRC-32 checksum value for chunk [0, 0x36ded): Mirror 1: 0x4d247c5f Mirror 8: 0x4d247c5f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ded: skipped 000667 1786064265.066199 read 0x00bccc thru 0x018d2d (0xd062 bytes) 000668 1786064265.096917 fallocate 0x007f20 thru 0x009e54 (0x1f35 bytes) 000669 1786064265.152109 trunc from 0x00dcb6 to 0x0194c5 (0xb810 bytes) 000670 1786064265.158840 fallocate 0x03a787 thru 0x03ffff (0x5879 bytes) 000671 1786064265.165914 punch from 0x033192 to 0x036dec (0x3c5b bytes) 000672 1786064265.172769 mapread 0x034dcc thru 0x036dec (0x2021 bytes) 000673 1786064265.209846 mapwrite 0x02c4f1 thru 0x03547b (0x8f8b bytes) 000674 1786064265.262605 punch from 0x030ab4 to 0x03427d (0x37ca bytes) 000675 1786064265.272563 trunc from 0x036ded to 0x026582 (0x1086a bytes) 000677 1786064265.320160 punch from 0x01d6e6 to 0x024c3c (0x7557 bytes) 000678 1786064265.327922 write 0x0184ef thru 0x01f7f9 (0x730b bytes) 000679 1786064265.341432 trunc from 0x018bc3 to 0x01c50f (0x394d bytes) 000680 1786064265.363108 mapread 0x01f8e7 thru 0x0215fb (0x1d15 bytes) 000681 1786064265.381145 punch from 0x01e938 to 0x01efe8 (0x6b1 bytes) 000683 1786064265.391015 mapwrite 0x01be9d thru 0x028b89 (0xcced bytes) 000685 1786064265.466523 write 0x009a32 thru 0x01261f (0x8bee bytes) 000686 1786064265.487648 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x28b8a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28b8a): Mirror 2: 0xe1242795 (stale) Mirror 3: 0xe1242795 (stale) Mirror 5: 0xe1242795 (stale) Mirror 7: 0xe1242795 (stale) Mirror 8: 0xfbfef1c9 (stale) CRC-32 checksum value for chunk [0, 0x28b8a): Mirror 1: 0x8a7a38f5 Mirror 9: 0x8a7a38f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28b8a: skipped 000687 1786064266.719692 read 0x015ab2 thru 0x019b5c (0x40ab bytes) 000688 1786064266.751207 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x28b8a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28b8a): Mirror 2: 0xe1242795 (stale) Mirror 3: 0xe1242795 (stale) Mirror 5: 0xe1242795 (stale) Mirror 7: 0xe1242795 (stale) Mirror 8: 0xfbfef1c9 (stale) CRC-32 checksum value for chunk [0, 0x28b8a): Mirror 1: 0x8a7a38f5 Mirror 9: 0x8a7a38f5 Mirror 10: 0x8a7a38f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28b8a: skipped 000689 1786064268.081955 mapwrite 0x010836 thru 0x0152fa (0x4ac5 bytes) 000691 1786064268.205474 punch from 0x01dc48 to 0x028b89 (0xaf42 bytes) 000692 1786064268.211804 mapread 0x00f186 thru 0x013270 (0x40eb bytes) 000694 1786064268.214910 fallocate 0x01e206 thru 0x01f826 (0x1621 bytes) 000695 1786064268.222486 write 0x033268 thru 0x036d6e (0x3b07 bytes) 000696 1786064268.229863 mapwrite 0x01c293 thru 0x02684d (0xa5bb bytes) 000697 1786064268.274064 fallocate 0x037952 thru 0x038dda (0x1489 bytes) 000698 1786064268.280228 fallocate 0x02b3a4 thru 0x03a804 (0xf461 bytes) 000699 1786064268.286872 write 0x03484f thru 0x03dde7 (0x9599 bytes) 000701 1786064268.295952 punch from 0x025e04 to 0x02930d (0x350a bytes) 000702 1786064268.304760 read 0x027dfb thru 0x02f773 (0x7979 bytes) 000703 1786064268.319830 punch from 0x00b5b4 to 0x00eb99 (0x35e6 bytes) 000704 1786064268.331746 fallocate 0x026e6c thru 0x031bd7 (0xad6c bytes) 000705 1786064268.340966 fallocate 0x02434a thru 0x032d1c (0xe9d3 bytes) 000706 1786064268.351551 mapread 0x02f118 thru 0x0352a9 (0x6192 bytes) 000707 1786064268.360625 write 0x017604 thru 0x017d04 (0x701 bytes) 000708 1786064268.362212 fallocate 0x0196e0 thru 0x01d170 (0x3a91 bytes) 000709 1786064268.371125 write 0x02b683 thru 0x03a001 (0xe97f bytes) 000710 1786064268.375538 fallocate 0x030179 thru 0x03c089 (0xbf11 bytes) 000711 1786064268.383570 fallocate 0x002c0a thru 0x00b18c (0x8583 bytes) 000712 1786064268.390813 fallocate 0x00cbd8 thru 0x016377 (0x97a0 bytes) 000713 1786064268.397229 punch from 0x03a607 to 0x03dde7 (0x37e1 bytes) 000714 1786064268.419127 fallocate 0x028a6c thru 0x02c817 (0x3dac bytes) 000715 1786064268.426270 trunc from 0x03dde8 to 0x020f6d (0x1ce7a bytes) 000716 1786064268.465168 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x20f6e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20f6e): Mirror 2: 0xb321e7f0 (stale) Mirror 3: 0xb321e7f0 (stale) Mirror 5: 0xb321e7f0 (stale) Mirror 7: 0xb321e7f0 (stale) Mirror 8: 0x9820f8bb (stale) Mirror 9: 0xc2ff7c02 (stale) Mirror 10: 0xc2ff7c02 (stale) CRC-32 checksum value for chunk [0, 0x20f6e): Mirror 1: 0x85291619 Mirror 11: 0x85291619 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20f6e: skipped 000717 1786064269.986552 punch from 0x018479 to 0x01a57e (0x2106 bytes) 000718 1786064270.055928 mapread 0x01636a thru 0x01c5b5 (0x624c bytes) 000719 1786064270.101023 punch from 0x009da1 to 0x0122e4 (0x8544 bytes) 000720 1786064270.115162 punch from 0x015748 to 0x01b125 (0x59de bytes) 000721 1786064270.130834 write 0x03c1bb thru 0x03ffff (0x3e45 bytes) 000723 1786064270.157557 write 0x00cd6c thru 0x017a5c (0xacf1 bytes) 000724 1786064270.180815 mapread 0x029432 thru 0x0386d8 (0xf2a7 bytes) 000725 1786064270.215019 trunc from 0x040000 to 0x03b243 (0x4dbc bytes) 000726 1786064270.268141 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3b244): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b244): Mirror 2: 0x8f2b992d (stale) Mirror 3: 0x8f2b992d (stale) Mirror 5: 0x8f2b992d (stale) Mirror 7: 0x8f2b992d (stale) Mirror 8: 0x4d00df21 (stale) Mirror 9: 0xb0acd1a7 (stale) Mirror 10: 0xb0acd1a7 (stale) Mirror 11: 0xb2456a09 (stale) CRC-32 checksum value for chunk [0, 0x3b244): Mirror 1: 0x1e3ee1c6 Mirror 12: 0x1e3ee1c6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b244: skipped 000727 1786064271.930887 mapwrite 0x03b4ce thru 0x03d91c (0x244f bytes) 000728 1786064272.078130 read 0x031337 thru 0x036a92 (0x575c bytes) 000730 1786064272.093377 read 0x03a676 thru 0x03d91c (0x32a7 bytes) 000731 1786064272.095040 mapread 0x028c2c thru 0x03776d (0xeb42 bytes) 000732 1786064272.117726 mapread 0x004a17 thru 0x01363b (0xec25 bytes) 000733 1786064272.129830 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d91d): Mirror 3: 0xf99362b1 (stale) Mirror 5: 0xf99362b1 (stale) Mirror 7: 0xf99362b1 (stale) Mirror 8: 0xe8a65e7c (stale) Mirror 9: 0xfa1479a6 (stale) Mirror 10: 0xfa1479a6 (stale) Mirror 11: 0xb1229ef1 (stale) Mirror 12: 0x66c5156a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d91d): Mirror 3: 0xf99362b1 (stale) Mirror 5: 0xf99362b1 (stale) Mirror 7: 0xf99362b1 (stale) Mirror 8: 0xe8a65e7c (stale) Mirror 9: 0xfa1479a6 (stale) Mirror 10: 0xfa1479a6 (stale) Mirror 11: 0xb1229ef1 (stale) Mirror 12: 0x66c5156a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d91d: skipped 000734 1786064272.873070 trunc from 0x00fd84 to 0x01930a (0x9587 bytes) 000736 1786064272.884154 trunc from 0x01a2dd to 0x02752c (0xd250 bytes) 000737 1786064272.890510 trunc from 0x03d91d to 0x021bb7 (0x1bd65 bytes) 000738 1786064272.920764 mapwrite 0x0259f5 thru 0x029561 (0x3b6d bytes) 000739 1786064272.989789 read 0x001715 thru 0x00ad06 (0x95f2 bytes) 000740 1786064272.999829 trunc from 0x00beb6 to 0x015105 (0x9250 bytes) 000741 1786064273.005419 mapwrite 0x034bdc thru 0x034edc (0x301 bytes) 000742 1786064273.054636 trunc from 0x034edd to 0x03d343 (0x8467 bytes) 000743 1786064273.075736 punch from 0x034bfb to 0x0377aa (0x2bb0 bytes) 000746 1786064273.084794 read 0x030069 thru 0x03741c (0x73b4 bytes) 000748 1786064273.097735 read 0x011e2f thru 0x020ca7 (0xee79 bytes) 000749 1786064273.099087 read 0x007dc5 thru 0x00e80c (0x6a48 bytes) 000750 1786064273.100101 write 0x037775 thru 0x03ffff (0x888b bytes) 000751 1786064273.104052 trunc from 0x040000 to 0x00d749 (0x328b6 bytes) 000753 1786064273.132888 mapread 0x00434a thru 0x008f93 (0x4c4a bytes) 000754 1786064273.135944 mapread 0x00aea4 thru 0x00b9cc (0xb29 bytes) 000755 1786064273.137894 punch from 0x0018d5 to 0x004fee (0x371a bytes) 000757 1786064273.150005 read 0x002cfe thru 0x004820 (0x1b23 bytes) 000758 1786064273.161023 read 0x00aa7e thru 0x00d749 (0x2ccc bytes) 000759 1786064273.162676 trunc from 0x00d74a to 0x0113b9 (0x3c70 bytes) 000760 1786064273.190591 read 0x00c848 thru 0x00fe69 (0x3622 bytes) 000761 1786064273.199252 write 0x029ce5 thru 0x032d80 (0x909c bytes) 000762 1786064273.202291 mapread 0x0042fb thru 0x010420 (0xc126 bytes) 000763 1786064273.207981 punch from 0x02149f to 0x0277a3 (0x6305 bytes) 000764 1786064273.213962 punch from 0x032329 to 0x032d80 (0xa58 bytes) 000766 1786064273.231750 write 0x01b351 thru 0x022d4f (0x79ff bytes) 000767 1786064273.247495 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32d81): Mirror 3: 0xfde8012c (stale) Mirror 5: 0xfde8012c (stale) Mirror 7: 0xfde8012c (stale) Mirror 8: 0xce1ce1a4 (stale) Mirror 10: 0x7ae53386 (stale) Mirror 11: 0xcae6ec83 (stale) Mirror 12: 0x9db4441f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32d81): Mirror 3: 0xfde8012c (stale) Mirror 5: 0xfde8012c (stale) Mirror 7: 0xfde8012c (stale) Mirror 8: 0xce1ce1a4 (stale) Mirror 10: 0x7ae53386 (stale) Mirror 11: 0xcae6ec83 (stale) Mirror 12: 0x9db4441f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32d81: skipped 000768 1786064274.011808 trunc from 0x0251aa to 0x028f01 (0x3d58 bytes) 000769 1786064274.027175 trunc from 0x032d81 to 0x030fd5 (0x1dab bytes) 000770 1786064274.067968 punch from 0x008a0c to 0x00f0ef (0x66e4 bytes) 000771 1786064274.084193 mapread 0x025570 thru 0x0259bb (0x44c bytes) 000773 1786064274.106121 trunc from 0x030fd6 to 0x01def5 (0x130e0 bytes) 000774 1786064274.155480 read 0x00f924 thru 0x0106b2 (0xd8f bytes) 000775 1786064274.177092 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1def6): Mirror 3: 0x824ccc2e (stale) Mirror 5: 0x824ccc2e (stale) Mirror 7: 0x824ccc2e (stale) Mirror 8: 0x915ca95e (stale) Mirror 11: 0x2e7123e9 (stale) Mirror 12: 0x901b5c43 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1def6): Mirror 3: 0x824ccc2e (stale) Mirror 5: 0x824ccc2e (stale) Mirror 7: 0x824ccc2e (stale) Mirror 8: 0x915ca95e (stale) Mirror 11: 0x2e7123e9 (stale) Mirror 12: 0x901b5c43 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1def6: skipped 000776 1786064274.754945 mapread 0x004aec thru 0x00b216 (0x672b bytes) 000777 1786064274.780441 read 0x00f954 thru 0x013e40 (0x44ed bytes) 000778 1786064274.786663 read 0x00b658 thru 0x01a170 (0xeb19 bytes) 000779 1786064274.799910 mapwrite 0x013f39 thru 0x01772e (0x37f6 bytes) 000780 1786064274.820756 mapwrite 0x01e356 thru 0x027a0e (0x96b9 bytes) 000781 1786064274.898015 trunc from 0x00c06e to 0x018d3e (0xccd1 bytes) 000782 1786064274.907014 fallocate 0x03bcc0 thru 0x03ffff (0x4340 bytes) 000784 1786064274.914628 fallocate 0x026807 thru 0x02e56e (0x7d68 bytes) 000785 1786064274.922671 trunc from 0x036e66 to 0x03ffff (0x919a bytes) 000786 1786064274.927759 mapread 0x0181a9 thru 0x026cdc (0xeb34 bytes) 000787 1786064274.954169 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x833aeab2 (stale) Mirror 5: 0x833aeab2 (stale) Mirror 7: 0x833aeab2 (stale) Mirror 8: 0x9e143d4a (stale) Mirror 11: 0xe31eb9a4 (stale) Mirror 12: 0x1bfbaff4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6f5ceb6e Mirror 13: 0x6f5ceb6e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000788 1786064276.388695 punch from 0x00c757 to 0x00f418 (0x2cc2 bytes) 000789 1786064276.462468 trunc from 0x03724a to 0x0372f2 (0xa9 bytes) 000790 1786064276.470876 trunc from 0x026907 to 0x03618d (0xf887 bytes) 000791 1786064276.477351 fallocate 0x01dad7 thru 0x028b74 (0xb09e bytes) 000792 1786064276.487927 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x833aeab2 (stale) Mirror 5: 0x833aeab2 (stale) Mirror 7: 0x833aeab2 (stale) Mirror 8: 0x9e143d4a (stale) Mirror 11: 0xe31eb9a4 (stale) Mirror 12: 0x1bfbaff4 (stale) Mirror 13: 0x6f5ceb6e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7affd0b5 Mirror 14: 0x7affd0b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000794 1786064277.799829 punch from 0x0302d0 to 0x03dec0 (0xdbf1 bytes) 000795 1786064277.872372 MIRROR_RESYNC 9 mirrors 000797 1786064280.653859 MIRROR_RESYNC 9 mirrors 000798 1786064281.411922 trunc from 0x0047f0 to 0x00d855 (0x9066 bytes) 000799 1786064281.445070 punch from 0x0106ea to 0x01a8af (0xa1c6 bytes) 000800 1786064281.452581 trunc from 0x026f63 to 0x02cb2d (0x5bcb bytes) 000801 1786064281.459052 mapwrite 0x03b035 thru 0x03ffff (0x4fcb bytes) 000802 1786064281.507142 trunc from 0x03259a to 0x038545 (0x5fac bytes) 000803 1786064281.513024 punch from 0x0354bb to 0x03ffff (0xab45 bytes) 000804 1786064281.524703 trunc from 0x03cce6 to 0x03ffff (0x331a bytes) 000806 1786064281.531982 write 0x02ea09 thru 0x03a280 (0xb878 bytes) 000807 1786064281.545474 mapread 0x01b32d thru 0x02ab75 (0xf849 bytes) 000810 1786064281.568176 mapread 0x008809 thru 0x014319 (0xbb11 bytes) 000811 1786064281.598111 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7affd0b5 (stale) Mirror 5: 0x7affd0b5 (stale) Mirror 7: 0x7affd0b5 (stale) Mirror 11: 0x7affd0b5 (stale) Mirror 12: 0x7affd0b5 (stale) Mirror 13: 0x7affd0b5 (stale) Mirror 14: 0x7affd0b5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7affd0b5 (stale) Mirror 5: 0x7affd0b5 (stale) Mirror 7: 0x7affd0b5 (stale) Mirror 11: 0x7affd0b5 (stale) Mirror 12: 0x7affd0b5 (stale) Mirror 13: 0x7affd0b5 (stale) Mirror 14: 0x7affd0b5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000812 1786064282.221030 mapread 0x010514 thru 0x018ed8 (0x89c5 bytes) 000813 1786064282.251866 MIRROR_RESYNC 8 mirrors 000814 1786064284.970659 trunc from 0x040000 to 0x02718f (0x18e70 bytes) 000815 1786064285.017209 fallocate 0x033008 thru 0x03c19b (0x9194 bytes) 000816 1786064285.032832 punch from 0x0191e6 to 0x02718f (0xdfaa bytes) 000817 1786064285.045291 fallocate 0x000461 thru 0x00d1ff (0xcd9f bytes) 000818 1786064285.055358 read 0x024367 thru 0x02718f (0x2e29 bytes) 000819 1786064285.064063 trunc from 0x027190 to 0x033c11 (0xca82 bytes) 000820 1786064285.096452 read 0x02a1a2 thru 0x033c11 (0x9a70 bytes) 000821 1786064285.119251 read 0x00f368 thru 0x01d44f (0xe0e8 bytes) 000822 1786064285.121209 fallocate 0x023914 thru 0x02c5e1 (0x8cce bytes) 000823 1786064285.128505 trunc from 0x0040c1 to 0x0136ee (0xf62e bytes) 000824 1786064285.143606 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33c12): Mirror 3: 0x26697ddf (stale) Mirror 7: 0x26697ddf (stale) Mirror 11: 0x26697ddf (stale) Mirror 12: 0x26697ddf (stale) Mirror 13: 0x26697ddf (stale) Mirror 14: 0x26697ddf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33c12): Mirror 3: 0x26697ddf (stale) Mirror 7: 0x26697ddf (stale) Mirror 11: 0x26697ddf (stale) Mirror 12: 0x26697ddf (stale) Mirror 13: 0x26697ddf (stale) Mirror 14: 0x26697ddf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33c12: skipped 000826 1786064285.903158 punch from 0x026bb3 to 0x028a7c (0x1eca bytes) 000827 1786064285.910624 trunc from 0x033c12 to 0x01306f (0x20ba2 bytes) 000828 1786064285.948503 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x13070): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13070): Mirror 3: 0x7eccbe57 (stale) Mirror 7: 0x7eccbe57 (stale) Mirror 11: 0x7eccbe57 (stale) Mirror 12: 0x7eccbe57 (stale) Mirror 13: 0x7eccbe57 (stale) Mirror 14: 0x7eccbe57 (stale) CRC-32 checksum value for chunk [0, 0x13070): Mirror 1: 0x7eccbe57 Mirror 15: 0x7eccbe57 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13070: skipped 000829 1786064287.078704 fallocate 0x028191 thru 0x02b0da (0x2f4a bytes) 000830 1786064287.137954 trunc from 0x003180 to 0x00635b (0x31dc bytes) 000831 1786064287.146637 punch from 0x000505 to 0x0076b3 (0x71af bytes) 000832 1786064287.154257 trunc from 0x00378e to 0x00801d (0x4890 bytes) 000833 1786064287.164564 MIRROR_RESYNC 8 mirrors 000834 1786064288.684037 punch from 0x00c149 to 0x01306f (0x6f27 bytes) 000836 1786064288.747537 mapwrite 0x0376c7 thru 0x03ebf6 (0x7530 bytes) 000837 1786064288.830424 MIRROR_RESYNC 8 mirrors 000838 1786064290.387841 mapread 0x029675 thru 0x034180 (0xab0c bytes) 000839 1786064290.428724 mapwrite 0x03a1b9 thru 0x03ffff (0x5e47 bytes) 000841 1786064290.544559 read 0x001cf9 thru 0x00de65 (0xc16d bytes) 000842 1786064290.563369 trunc from 0x0099f5 to 0x00f7a2 (0x5dae bytes) 000843 1786064290.569545 read 0x034e30 thru 0x03ffff (0xb1d0 bytes) 000844 1786064290.570754 fallocate 0x03d716 thru 0x03ffff (0x28ea bytes) 000845 1786064290.578620 punch from 0x01a058 to 0x0248d1 (0xa87a bytes) 000846 1786064290.584751 read 0x01114b thru 0x019bc7 (0x8a7d bytes) 000847 1786064290.598732 fallocate 0x0050b2 thru 0x012210 (0xd15f bytes) 000848 1786064290.606219 write 0x02a49e thru 0x039760 (0xf2c3 bytes) 000849 1786064290.609724 trunc from 0x02635c to 0x032724 (0xc3c9 bytes) 000850 1786064290.629025 write 0x0280f8 thru 0x02ba5d (0x3966 bytes) 000851 1786064290.641190 mapwrite 0x03abdf thru 0x03e55c (0x397e bytes) 000852 1786064290.664644 mapwrite 0x007dba thru 0x0105d4 (0x881b bytes) 000853 1786064290.692521 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x98944a9b (stale) Mirror 11: 0x98944a9b (stale) Mirror 12: 0x98944a9b (stale) Mirror 13: 0x98944a9b (stale) Mirror 14: 0x98944a9b (stale) Mirror 15: 0x98944a9b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x98944a9b (stale) Mirror 11: 0x98944a9b (stale) Mirror 12: 0x98944a9b (stale) Mirror 13: 0x98944a9b (stale) Mirror 14: 0x98944a9b (stale) Mirror 15: 0x98944a9b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000854 1786064291.387150 trunc from 0x03704f to 0x0397ae (0x2760 bytes) 000855 1786064291.398637 punch from 0x02de63 to 0x03d75f (0xf8fd bytes) 000856 1786064291.406541 read 0x03cd22 thru 0x03ffff (0x32de bytes) 000858 1786064291.416069 trunc from 0x040000 to 0x01f1e8 (0x20e17 bytes) 000860 1786064291.440643 fallocate 0x0049bc thru 0x00c8d4 (0x7f19 bytes) 000861 1786064291.449243 read 0x01d2c2 thru 0x01f1e8 (0x1f27 bytes) 000862 1786064291.459971 read 0x0181d7 thru 0x01f1e8 (0x7012 bytes) 000863 1786064291.461087 read 0x015a7e thru 0x01f1e8 (0x976b bytes) 000864 1786064291.462351 trunc from 0x01f1e9 to 0x03be7a (0x1cc92 bytes) 000865 1786064291.481554 punch from 0x01f76e to 0x02a9c0 (0xb253 bytes) 000866 1786064291.487697 read 0x02bd14 thru 0x03475e (0x8a4b bytes) 000868 1786064291.507265 fallocate 0x03bce9 thru 0x03ffff (0x4317 bytes) 000869 1786064291.512862 punch from 0x00be21 to 0x017301 (0xb4e1 bytes) 000870 1786064291.520710 mapwrite 0x00c64b thru 0x0135d0 (0x6f86 bytes) 000871 1786064291.599916 read 0x02bf30 thru 0x032a2b (0x6afc bytes) 000873 1786064291.602125 read 0x02f787 thru 0x02fe0f (0x689 bytes) 000874 1786064291.603744 mapwrite 0x02bc87 thru 0x02d19e (0x1518 bytes) 000875 1786064291.620614 trunc from 0x040000 to 0x0321f9 (0xde06 bytes) 000877 1786064291.645565 punch from 0x003686 to 0x00c423 (0x8d9e bytes) 000878 1786064291.651895 fallocate 0x02f4e6 thru 0x03cd37 (0xd852 bytes) 000879 1786064291.658946 read 0x02c6c7 thru 0x032abe (0x63f8 bytes) 000880 1786064291.660096 mapread 0x01be42 thru 0x021136 (0x52f5 bytes) 000881 1786064291.661759 mapwrite 0x02d024 thru 0x03bc48 (0xec25 bytes) 000882 1786064291.704473 mapwrite 0x010613 thru 0x0109ab (0x399 bytes) 000883 1786064291.720071 read 0x03aa4e thru 0x03cd37 (0x22ea bytes) 000884 1786064291.721486 write 0x01f8b9 thru 0x023c6f (0x43b7 bytes) 000885 1786064291.727580 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3cd38): Mirror 3: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd38): Mirror 3: 0xf21273bb (stale) Mirror 11: 0xf21273bb (stale) Mirror 12: 0xf21273bb (stale) Mirror 13: 0xf21273bb (stale) Mirror 14: 0xf21273bb (stale) Mirror 15: 0xf21273bb (stale) CRC-32 checksum value for chunk [0, 0x3cd38): Mirror 1: 0x2fb94501 Mirror 16: 0x2fb94501 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd38: skipped 000886 1786064293.382000 mapwrite 0x027740 thru 0x032a70 (0xb331 bytes) 000887 1786064293.627934 mapread 0x002231 thru 0x00d9cf (0xb79f bytes) 000888 1786064293.653444 trunc from 0x03cd38 to 0x00574b (0x375ec bytes) 000889 1786064293.713638 read 0x0037cc thru 0x0049bd (0x11f2 bytes) 000890 1786064293.717752 trunc from 0x004cf3 to 0x00574b (0xa59 bytes) 000891 1786064293.729017 mapwrite 0x004fa8 thru 0x008dd0 (0x3e29 bytes) 000892 1786064293.815435 trunc from 0x008b22 to 0x008dd0 (0x2af bytes) 000893 1786064293.823118 trunc from 0x008dd1 to 0x03a629 (0x31859 bytes) 000894 1786064293.859245 mapread 0x00c4e9 thru 0x011517 (0x502f bytes) 000895 1786064293.883256 punch from 0x0110e8 to 0x0201d0 (0xf0e9 bytes) 000896 1786064293.893380 write 0x00e609 thru 0x017b87 (0x957f bytes) 000897 1786064293.904059 fallocate 0x03e67d thru 0x03ffff (0x1983 bytes) 000898 1786064293.911355 trunc from 0x040000 to 0x03f520 (0xadf bytes) 000899 1786064293.943579 trunc from 0x00e0b5 to 0x01146b (0x33b7 bytes) 000900 1786064293.968956 trunc from 0x004f70 to 0x006546 (0x15d7 bytes) 000901 1786064293.983380 trunc from 0x0225bd to 0x030c5f (0xe6a3 bytes) 000902 1786064293.992518 punch from 0x0209f1 to 0x023fa1 (0x35b1 bytes) 000903 1786064293.999018 mapread 0x00de51 thru 0x016c15 (0x8dc5 bytes) 000904 1786064294.010591 punch from 0x003729 to 0x01179f (0xe077 bytes) 000905 1786064294.020973 fallocate 0x031ca4 thru 0x03ae0f (0x916c bytes) 000906 1786064294.026610 mapread 0x005091 thru 0x005868 (0x7d8 bytes) 000907 1786064294.039551 read 0x033cc0 thru 0x03f520 (0xb861 bytes) 000908 1786064294.055092 read 0x01e872 thru 0x02e1ed (0xf97c bytes) 000909 1786064294.056614 punch from 0x00a240 to 0x00ccf3 (0x2ab4 bytes) 000910 1786064294.071256 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3f521): Mirror 3: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f521): Mirror 3: 0x5a40eea9 (stale) Mirror 11: 0x5a40eea9 (stale) Mirror 12: 0x5a40eea9 (stale) Mirror 13: 0x5a40eea9 (stale) Mirror 14: 0x5a40eea9 (stale) Mirror 15: 0x5a40eea9 (stale) Mirror 16: 0x137efe07 (stale) CRC-32 checksum value for chunk [0, 0x3f521): Mirror 1: 0x6d645ba4 Mirror 17: 0x6d645ba4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f521: skipped 000911 1786064295.235250 trunc from 0x03f521 to 0x0150d0 (0x2a450 bytes) 000912 1786064295.299039 fallocate 0x0330e8 thru 0x03c910 (0x9829 bytes) 000913 1786064295.305707 read 0x01bf05 thru 0x020c98 (0x4d94 bytes) 000914 1786064295.316622 trunc from 0x02fc89 to 0x03035b (0x6d3 bytes) 000915 1786064295.322922 fallocate 0x030399 thru 0x037e69 (0x7ad1 bytes) 000916 1786064295.328574 mapwrite 0x0379ae thru 0x0388d9 (0xf2c bytes) 000917 1786064295.354974 trunc from 0x03c911 to 0x01211d (0x2a7f3 bytes) 000918 1786064295.382105 fallocate 0x03dc4e thru 0x03ffff (0x23b2 bytes) 000919 1786064295.395914 MIRROR_RESYNC 9 mirrors 000920 1786064297.691324 fallocate 0x028b4a thru 0x034976 (0xbe2d bytes) 000921 1786064297.736223 write 0x00abf2 thru 0x00ddb2 (0x31c1 bytes) 000922 1786064297.748571 mapwrite 0x0369ac thru 0x03ffff (0x9654 bytes) 000923 1786064297.800236 write 0x02adf3 thru 0x038cb4 (0xdec2 bytes) 000924 1786064297.809269 trunc from 0x040000 to 0x00f97a (0x30685 bytes) 000925 1786064297.841736 mapread 0x00c876 thru 0x00e294 (0x1a1f bytes) 000926 1786064297.856046 read 0x000ba9 thru 0x00356a (0x29c2 bytes) 000927 1786064297.866223 fallocate 0x02309b thru 0x02ed58 (0xbcbe bytes) 000928 1786064297.874368 trunc from 0x004895 to 0x012bb6 (0xe322 bytes) 000929 1786064297.882018 fallocate 0x002459 thru 0x00ca58 (0xa600 bytes) 000930 1786064297.889853 fallocate 0x027c56 thru 0x02cc4d (0x4ff8 bytes) 000931 1786064297.896293 fallocate 0x02d7ae thru 0x03acaf (0xd502 bytes) 000932 1786064297.903518 fallocate 0x02081a thru 0x02d4e8 (0xcccf bytes) 000933 1786064297.911076 trunc from 0x02ed59 to 0x0146be (0x1a69a bytes) 000934 1786064297.945134 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x146bf): Mirror 3: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x146bf): Mirror 3: 0x33017c3a (stale) Mirror 11: 0x33017c3a (stale) Mirror 12: 0x33017c3a (stale) Mirror 13: 0x33017c3a (stale) Mirror 14: 0x33017c3a (stale) Mirror 15: 0x33017c3a (stale) Mirror 16: 0x33017c3a (stale) Mirror 17: 0x33017c3a (stale) CRC-32 checksum value for chunk [0, 0x146bf): Mirror 1: 0x24783d27 Mirror 18: 0x24783d27 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x146bf: skipped 000935 1786064299.208516 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x146bf): Mirror 3: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x146bf): Mirror 3: 0x33017c3a (stale) Mirror 11: 0x33017c3a (stale) Mirror 12: 0x33017c3a (stale) Mirror 13: 0x33017c3a (stale) Mirror 14: 0x33017c3a (stale) Mirror 15: 0x33017c3a (stale) Mirror 16: 0x33017c3a (stale) Mirror 17: 0x33017c3a (stale) CRC-32 checksum value for chunk [0, 0x146bf): Mirror 1: 0x24783d27 Mirror 18: 0x24783d27 Mirror 19: 0x24783d27 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x146bf: skipped 000936 1786064300.471997 MIRROR_RESYNC 11 mirrors 000937 1786064302.113929 trunc from 0x0146bf to 0x018879 (0x41bb bytes) 000938 1786064302.184703 write 0x01f72e thru 0x0241fe (0x4ad1 bytes) 000939 1786064302.187081 mapwrite 0x032c5b thru 0x03b9df (0x8d85 bytes) 000940 1786064302.277223 fallocate 0x024f81 thru 0x03492d (0xf9ad bytes) 000941 1786064302.293695 trunc from 0x033d3b to 0x0385ad (0x4873 bytes) 000942 1786064302.309104 fallocate 0x0038e4 thru 0x0098ac (0x5fc9 bytes) 000943 1786064302.319882 read 0x0112b1 thru 0x013c15 (0x2965 bytes) 000944 1786064302.333523 mapread 0x01e9fc thru 0x0253dd (0x69e2 bytes) 000945 1786064302.358165 mapread 0x01ecd4 thru 0x0289e4 (0x9d11 bytes) 000946 1786064302.383030 fallocate 0x00bb86 thru 0x017937 (0xbdb2 bytes) 000947 1786064302.395529 fallocate 0x01ab57 thru 0x029364 (0xe80e bytes) 000949 1786064302.407092 trunc from 0x031338 to 0x03930d (0x7fd6 bytes) 000950 1786064302.425184 trunc from 0x03b9e0 to 0x01f25f (0x1c780 bytes) 000951 1786064302.455120 fallocate 0x003f01 thru 0x00cf99 (0x9099 bytes) 000952 1786064302.472209 read 0x0100d4 thru 0x01824f (0x817c bytes) 000953 1786064302.503568 punch from 0x01c22e to 0x01f25f (0x3032 bytes) 000954 1786064302.514144 fallocate 0x0272d0 thru 0x031b68 (0xa899 bytes) 000955 1786064302.524695 trunc from 0x012bd4 to 0x015484 (0x28b1 bytes) 000956 1786064302.533266 mapread 0x0032cc thru 0x00a8e4 (0x7619 bytes) 000957 1786064302.539160 trunc from 0x0001da to 0x00fa4d (0xf874 bytes) 000958 1786064302.552948 write 0x0123f6 thru 0x01e2c7 (0xbed2 bytes) 000959 1786064302.571798 fallocate 0x0223d7 thru 0x028a64 (0x668e bytes) 000960 1786064302.579850 trunc from 0x01f260 to 0x03c856 (0x1d5f7 bytes) 000961 1786064302.622321 trunc from 0x03c857 to 0x012a8a (0x29dcc bytes) 000962 1786064302.654321 read 0x00dd9d thru 0x011ca1 (0x3f05 bytes) 000963 1786064302.672150 mapwrite 0x03088c thru 0x03789b (0x7010 bytes) 000964 1786064302.735704 punch from 0x018d61 to 0x021068 (0x8308 bytes) 000965 1786064302.752404 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x120:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3789c): Mirror 3: 0xb08205fc (stale) Mirror 11: 0xb08205fc (stale) Mirror 12: 0xb08205fc (stale) Mirror 13: 0xb08205fc (stale) Mirror 14: 0xb08205fc (stale) Mirror 15: 0xb08205fc (stale) Mirror 17: 0xb08205fc (stale) Mirror 18: 0xb08205fc (stale) Mirror 19: 0xb08205fc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3789c): Mirror 3: 0xb08205fc (stale) Mirror 11: 0xb08205fc (stale) Mirror 12: 0xb08205fc (stale) Mirror 13: 0xb08205fc (stale) Mirror 14: 0xb08205fc (stale) Mirror 15: 0xb08205fc (stale) Mirror 17: 0xb08205fc (stale) Mirror 18: 0xb08205fc (stale) Mirror 19: 0xb08205fc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3789c: skipped 000966 1786064303.457202 trunc from 0x016883 to 0x01a504 (0x3c82 bytes) 000967 1786064303.464071 write 0x0015b9 thru 0x009e97 (0x88df bytes) 000968 1786064303.484814 read 0x001379 thru 0x0105f2 (0xf27a bytes) 000969 1786064303.495644 write 0x0232ec thru 0x025db3 (0x2ac8 bytes) 000970 1786064303.507103 mapwrite 0x02303b thru 0x02f944 (0xc90a bytes) 000971 1786064303.553895 fallocate 0x005ed7 thru 0x00e1ea (0x8314 bytes) 000972 1786064303.563773 read 0x00c2ff thru 0x01bdc1 (0xfac3 bytes) 000973 1786064303.590957 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3789c): Mirror 3: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3789c): Mirror 3: 0xb08205fc (stale) Mirror 11: 0xb08205fc (stale) Mirror 12: 0xb08205fc (stale) Mirror 13: 0xb08205fc (stale) Mirror 14: 0xb08205fc (stale) Mirror 15: 0xb08205fc (stale) Mirror 17: 0xb08205fc (stale) Mirror 18: 0xb08205fc (stale) Mirror 19: 0xb08205fc (stale) CRC-32 checksum value for chunk [0, 0x3789c): Mirror 1: 0xef366cc7 Mirror 20: 0xef366cc7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3789c: skipped 000974 1786064304.956037 read 0x02f413 thru 0x0351fb (0x5de9 bytes) 000975 1786064304.977853 mapread 0x00b770 thru 0x00da29 (0x22ba bytes) 000977 1786064305.038710 mapread 0x021619 thru 0x0294b8 (0x7ea0 bytes) 000978 1786064305.069609 trunc from 0x03789c to 0x029e3d (0xda5e bytes) 000979 1786064305.149885 trunc from 0x029e3e to 0x02242d (0x7a10 bytes) 000980 1786064305.172654 mapwrite 0x003228 thru 0x00ebbb (0xb994 bytes) 000981 1786064305.240710 MIRROR_RESYNC 11 mirrors 000982 1786064307.306788 fallocate 0x005951 thru 0x011af7 (0xc1a7 bytes) 000983 1786064307.371113 punch from 0x0214d7 to 0x02242d (0xf57 bytes) 000985 1786064307.379042 trunc from 0x02242e to 0x00470a (0x1dd23 bytes) 000987 1786064307.435170 read 0x0022ef thru 0x00470a (0x241c bytes) 000988 1786064307.459086 trunc from 0x00470b to 0x032b65 (0x2e45b bytes) 000989 1786064307.489752 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x32b66): Mirror 3: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32b66): Mirror 3: 0xae3c4237 (stale) Mirror 11: 0xae3c4237 (stale) Mirror 12: 0xae3c4237 (stale) Mirror 13: 0xae3c4237 (stale) Mirror 14: 0xae3c4237 (stale) Mirror 15: 0xae3c4237 (stale) Mirror 17: 0xae3c4237 (stale) Mirror 18: 0xae3c4237 (stale) Mirror 19: 0xae3c4237 (stale) Mirror 20: 0xae3c4237 (stale) CRC-32 checksum value for chunk [0, 0x32b66): Mirror 1: 0xb24a0475 Mirror 21: 0xb24a0475 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32b66: skipped 000991 1786064308.896215 fallocate 0x03aae1 thru 0x03ffff (0x551f bytes) 000992 1786064308.965073 mapread 0x015236 thru 0x01b734 (0x64ff bytes) 000993 1786064309.000936 trunc from 0x032b66 to 0x017e74 (0x1acf1 bytes) 000995 1786064309.068734 mapread 0x003a96 thru 0x009808 (0x5d73 bytes) 000996 1786064309.088900 read 0x0135ab thru 0x017e74 (0x48ca bytes) 000997 1786064309.090450 mapwrite 0x026ea3 thru 0x036a2c (0xfb8a bytes) 000998 1786064309.165324 trunc from 0x036a2d to 0x021241 (0x157eb bytes) 001000 1786064309.201026 mapread 0x01c212 thru 0x021241 (0x5030 bytes) All operations completed A-OK! PASS 70a (139s) == sanity-flr test 71: check mirror extend parent fid ==== 20:58:35 (1786064315) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x168:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 20:58:47 (1786064327) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.257991 s, 8.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876264 s, 12.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 20:58:57 (1786064337) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre2 Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 1514880 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6603616 .. Extending file size to 5879424 .. Extending file size to 8462976 .. Extending file size to 8920032 .. Extending file size to 8022496 .. Extending file size to 8248416 .. Extending file size to 8874656 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 61171 61172 61175 61177 61179 failed 192.168.204.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg405-client.virtnet /mnt/lustre2 (opts:) 192.168.204.105@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg405-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 21:00:22 (1786064422) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre2 Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5025504 .. Extending file size to 6663296 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7801632 .. Extending file size to 5905120 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7697440 .. Extending file size to 8012384 .. Extending file size to 8773792 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8328416 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62700 62701 62702 62704 62707 done 192.168.204.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg405-client.virtnet /mnt/lustre2 (opts:) 192.168.204.105@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg405-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 21:01:44 (1786064504) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2208 1269096 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18460 3588560 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24032 3582988 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42492 7171548 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2208 1269096 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18460 3588560 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24032 3582988 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42492 7171548 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.086438 s, 24.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.276565 s, 7.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x30e30e2d (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg405-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 21:02:06 (1786064526) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.154573 s, 13.6 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 21:02:15 (1786064535) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x62:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 21:02:24 (1786064544) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.112368 s, 18.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1b0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1af:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1b0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 21:02:33 (1786064553) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.142175 s, 14.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1b1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1b2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1b1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1b1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1b2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1b1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1b2:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 21:02:44 (1786064564) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.125536 s, 16.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 21:03:20 (1786064600) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg405-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112537 s, 9.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1b3:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1b4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1b3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1b3:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1b4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1b3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1b4:0x0] } PASS 204d (54s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 21:04:14 (1786064654) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg405-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1b5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1b6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.125505 s, 16.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1b5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1b6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1b5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1b5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1b6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1b5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (57s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 21:05:11 (1786064711) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg405-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg405-server: Pool lustre.test_204f created Adding targets to pool oleg405-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg405-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.129983 s, 16.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg405-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg405-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg405-server: Pool lustre.test_204f destroyed PASS 204f (84s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 21:06:35 (1786064795) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 21:06:45 (1786064805) PASS 205b (11s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 21:06:56 (1786064816) oleg405-server: Pool lustre.test_206 created Adding targets to pool oleg405-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg405-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg405-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg405-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg405-server: Pool lustre.test_206 destroyed PASS 206 (31s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 21:07:28 (1786064848) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104618 s, 10.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370576 s, 28.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xe3a3bd30 Mirror 3: 0xe3a3bd30 PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 21:07:41 (1786064861) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 21:07:46 (1786064866) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 21:07:51 (1786064871) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0760741 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x15d0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1190000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1190000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1190000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x980000 Read this (4 bytes): BBBB 192.168.204.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg405-client.virtnet /mnt/lustre2 (opts:) PASS 209a (44s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 21:08:36 (1786064916) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0160134 s, 512 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 21:08:45 (1786064925) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078373 s, 13.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7f lmm_fid: [0x200000405:0x7f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7f lmm_fid: [0x200000405:0x7f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11b:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611263 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-06 21:08:47.000000000 -0400 Modify: 2026-08-06 21:08:47.000000000 -0400 Change: 2026-08-06 21:08:49.000000000 -0400 Birth: 2026-08-06 21:08:48.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 21:08:56 (1786064936) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802425 s, 13.1 MB/s oleg405-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg405-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg405-server: O/280000401/d25/139: File not found by ext2_lookup PASS 210b (27s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 21:09:22 (1786064962) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.225983 s, 181 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.025542 s, 160 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 6 21:09 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x84:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 6 21:09 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 21:09:32 (1786064972) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (11s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 21:09:44 (1786064984) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 2184 sec =========== 21:09:55 (1786064995) === sanity-flr: start cleanup 21:09:57 (1786064997) === === sanity-flr: finish cleanup 21:10:07 (1786065007) ===